{"id":370,"date":"2020-03-24T13:37:11","date_gmt":"2020-03-24T19:37:11","guid":{"rendered":"https:\/\/briancroxall.net\/w20dh\/?page_id=370"},"modified":"2020-03-24T13:49:25","modified_gmt":"2020-03-24T19:49:25","slug":"topic-modeling-step-by-step","status":"publish","type":"page","link":"https:\/\/briancroxall.net\/w20dh\/topic-modeling-step-by-step\/","title":{"rendered":"Topic Modeling Step-by-Step"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">So you want to be a <del>hero<\/del> topic modeler? It turns out it&#8217;s not that difficult, thanks to the work those who created and maintain MALLET and the team behind&nbsp;<em>The Programming Historian<\/em>&#8216;s <a href=\"https:\/\/programminghistorian.org\/lessons\/topic-modeling-and-mallet\">lesson on topic modeling<\/a>&nbsp;(see more in credits below).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The 7 Steps of Highly Effective Topic Modelers<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. Get the JDK<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Download and install v8u241 of the <a href=\"https:\/\/www.oracle.com\/java\/technologies\/javase-jdk8-downloads.html\">Java Development Kit<\/a> (JDK). Make sure you choose the right version for your platform. If you&#8217;re a Windows user, make sure you download the x64 version and <em>not <\/em>the x86 version.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"436\" src=\"https:\/\/briancroxall.net\/w20dh\/wp-content\/uploads\/2020\/03\/Java-SE-Development-Kit-8-Downloads-1024x436.png\" alt=\"\" class=\"wp-image-373\" srcset=\"https:\/\/briancroxall.net\/w20dh\/wp-content\/uploads\/2020\/03\/Java-SE-Development-Kit-8-Downloads-1024x436.png 1024w, https:\/\/briancroxall.net\/w20dh\/wp-content\/uploads\/2020\/03\/Java-SE-Development-Kit-8-Downloads-300x128.png 300w, https:\/\/briancroxall.net\/w20dh\/wp-content\/uploads\/2020\/03\/Java-SE-Development-Kit-8-Downloads-768x327.png 768w, https:\/\/briancroxall.net\/w20dh\/wp-content\/uploads\/2020\/03\/Java-SE-Development-Kit-8-Downloads.png 1458w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">2. Get MALLET<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"http:\/\/mallet.cs.umass.edu\">MALLET<\/a> is the&nbsp;MAchine Learning for LanguagE Toolkit, which was developed at U Mass Amherst by Andrew McCallum and his grad students. <a href=\"http:\/\/mallet.cs.umass.edu\/download.php\">Download<\/a> the right zip file for your computer. What you do next depends on your operating system.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Windows PC<\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>Unzip MALLET into your C: directory. This is important: it cannot be anywhere else. You will then have a directory called\u00a0<code>c:\\mallet-2.0.8<\/code>\u00a0or similar.<\/li><li>MALLET uses an environment variable to tell the computer where to find all the various components of its processes when it is running. It\u2019s rather like a shortcut for the program. A programmer cannot know exactly where every user will install a program, so the programmer creates a variable in the code that will always stand in for that location. We tell the computer, once, where that location is by setting the environment variable. If you moved the program to a new location, you\u2019d have to change the variable.<\/li><li>To create an environment variable in Windows, click on your\u00a0<code>Start Menu -> Control Panel -> System -> Advanced System Settings<\/code>. Click new and type\u00a0<code>MALLET_HOME<\/code>\u00a0in the variable name box. It must be like this \u2013 all caps, with an underscore \u2013 since that is the shortcut that the programmer built into the program and all of its subroutines. Then type the exact path (location) of where you unzipped MALLET in the variable value, e.g.,\u00a0<code>c:\\mallet-2.0.8<\/code>.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Mac<\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>Unzip the directory somewhere on your computer. It might be easiest to put it in your Documents folder. In that folder we\u2019ll create a new folder called \u201ctm\u201d for topic modeling (<code>mkdir<\/code>\u00a0on the command line). <\/li><li>Unzip the file in that folder.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Check commands<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Go to the <a href=\"https:\/\/programminghistorian.org\/en\/lessons\/intro-to-bash\">command line<\/a> on your computer (either the Terminal app on a Mac or the Command Prompt on Windows). Navigate to the directory where you installed MALLET.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Type\u00a0<code>.\/bin\/mallet<\/code>\u00a0(Mac) or\u00a0<code>bin\\mallet<\/code> (PC) to see the commands that you can run.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Increase MALLET memory (optional)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">MALLET will work better if you give it more memory than it is allowed by default.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Go to the&nbsp;<code>bin<\/code>&nbsp;directory in the MALLET folder.<\/li><li>Open the&nbsp;<code>mallet<\/code>&nbsp;file in a text editor.<\/li><li>Change the&nbsp;<code>MEMORY=1g<\/code>&nbsp;to a higher number to make more use of your RAM. Naturally, you can&#8217;t use more memory than you have. For example, I have 16GB of RAM, and I have set MALLET to use 8GB.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. Create a MALLET corpus<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now you&#8217;re almost ready to start topic modeling. This is a two-step process. The first step is assembling all of your documents and then putting them into a single file for MALLET to read. It will have the extension <code>.mallet<\/code>. I&#8217;ll refer to this file as a MALLET corpus.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, bring all of your texts into one directory somewhere on your computer. Second, from the command line:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>make sure you\u2019re in the MALLET directory<\/li><li>run one of the following commands based on your operating system:<\/li><li>on a Mac:&nbsp;<code>.\/bin\/mallet import-dir --input \u201cdirectory\u201d --output \u201cname\u201d.mallet --keep-sequence<\/code><\/li><li>on a PC: <code>bin\\mallet import-dir --input \"directory\" --output \"name\".mallet --keep-sequence<\/code>\n<ul><li><code>.\/bin\/mallet<\/code>&nbsp;invokes MALLET<\/li><li><code>import-dir<\/code>&nbsp;tells it that you\u2019re importing a directory<\/li><li><code>--input<\/code>&nbsp;plus the directory location tells it what text files you want to pull from<\/li><li><code>--output<\/code>&nbsp;tells it what the MALLET corpus should be named<\/li><li><code>--keep-sequence<\/code>: \u201cThis option preserves the document as a sequence of word features, rather than a vector of word feature counts. Use this option for sequence labeling tasks. The MALLET topic modeling toolkit also requires feature sequences rather than feature vectors.\u201d<\/li><\/ul>\n<\/li><li>Check that you have a new&nbsp;<code>.mallet<\/code>&nbsp;file in your directory with the&nbsp;<code>ls<\/code>&nbsp;command.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Optional Commands for Stopwords<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">MALLET comes with a stopword list that you can call when creating the MALLET corpus. To do this, add the command <code>--remove-stopwords<\/code> to the import command. To see what terms are included in the default MALLET stopwords list, go to the MALLET directory &gt; stoplists &gt; en.txt.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You\u00a0<em>cannot<\/em> add words to the MALLET stopwords file. (Well, you <em>can<\/em>, but it won&#8217;t affect what MALLET does as the list is just there as a reference for an internal function.) Instead, you can create another text file (in TextEdit on Mac or Notepad on Windows) with one stopword per line. You should name and save this file to your MALLET directory (e.g., <code>extra-stopwords.txt<\/code>). When creating your MALLET corpus, you would use both <code>--remove-stopwords<\/code> and a new flag, <code>--extra-stopwords [filename]<\/code> (e.g., <code>--remove-stopwords --extra-stopwords extra-stopwords.txt<\/code>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Remember, of course, that using stop words will radically change your output. Doing so can be good <em>or<\/em> bad; the one thing for certain is that it is never simply neutral.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. Train Your Topics<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now that we have a MALLET corpus, we are ready to start&nbsp;the unsupervised machine learning. This is referred to as &#8220;training&#8221; your topics.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">From the command line, make sure that you&#8217;re (still) in your MALLET directory.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Run the following command on a Mac:&nbsp;<code>.\/bin\/mallet train-topics --input \u201cname\u201d.mallet --num-topics \"X\" --output-topic-keys \u201cname\u201d-keys.txt --output-doc-topics \u201cname\u201d-composition.txt<\/code>. For a PC, use the same command but start it with <code>bin\\mallet train-topics...<\/code>.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>.\/bin\/mallet<\/code>\u00a0tells it what file will run the command<\/li><li><code>train-topics<\/code>\u00a0is the command that you\u2019re running<\/li><li><code>--input<\/code>\u00a0plus the file name that follows it is the MALLET file you\u2019ll perform your train your models on<\/li><li><code>--num-topics<\/code>\u00a0plus the integer you put after it is the number of topics you have the model find. Default is 10. <ul><li>The number of topics that you should use is perhaps the most vexing question for topic modeling. Here&#8217;s a useful passage from Shawn Graham, Scott Weingart, and Ian Milligan&#8217;s,\u00a0<a href=\"https:\/\/programminghistorian.org\/lessons\/topic-modeling-and-mallet#typing-in-mallet-commands\"><em>Programming Historian<\/em> tutorial<\/a>:\u00a0 &#8220;How do you know the number of topics to search for? Is there a natural number of topics? What we have found is that one has to run the train-topics with varying numbers of topics to see how the composition file breaks down. If we end up with the majority of our original texts all in a very limited number of topics, then we take that as a signal that we need to increase the number of topics.&#8221; <\/li><\/ul><\/li><li><code>--output<\/code> Two different files that will contain our results from the training process. See below for more details.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Optional Commands<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Here is another command that you can use while training your topics:&nbsp;<code>--num-iterations \"X\" --optimize-interval \u201cX\u201d<\/code>.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>--num-iterations<\/code>&nbsp;plus the integer that follows tells your computer how many times it should run through the process before it yields the final models.<\/li><li><code>--optimize-interval<\/code>&nbsp;plus the integer you put after allows the model to learn from itself as it goes along. I find that it gets better results to include this. You need to set this to an integer. Optimization should take place every 10 intervals, <a href=\"http:\/\/mallet.cs.umass.edu\/topics.php\">according to the MALLET website<\/a>.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6. Record Your Work<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Like regular close reading, there is definitely some art to the process of distant reading. One of the advantages that the latter has over the former, however, is that it can be a lot simpler to record your process so your work could be replicated. That said, once you do a series of topic models, it can be really easy to lose track of what you&#8217;ve done. Therefore it&#8217;s important to keep a record as you work.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Move your two output files to another directory. You might want to name the folder with something helpful like &#8220;peanuts-nostop-10topics.&#8221;<\/li><li>You should also create a note of what you did to create these files. Create a text file (either from your GUI or from the command line using <code>touch \"name\".txt<\/code>).\u00a0Go to the terminal and copy the commands that you used to create the file and then paste them into the text file.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7. Interpret Your Results<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We created two files from our process:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>topic-keys.txt: a text document showing you what the top \/ &#8220;key&#8221; words are for each topic.<\/li><li>composition.txt: outputs a text file indicating the breakdown, by percentage, of each topic within each original text file you imported. This is normally best viewed in Excel.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Take a look at the topic-keys. See if there are multiple topics that seem to use the same terms. If so, you might want to re-run the model with fewer topics. If there doesn&#8217;t seem to be coherence among the topics, you might need to re-run the model with more topics. Consider whether there are terms that seem to be throwing off or clogging the model, including stopwords or character names. If you want to remove these, you will have to return to Step 4 to remove them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Take a look at the composition in a spreadsheet like Excel. Find a topic that you&#8217;re interested in, and look for the number of the topic in the column at the top of the page. Sort the spreadsheet by that column and look at which documents have the highest percentage of being composed from that topic. Are the results expected? Do the documents seem to belong together?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Other Resources<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">David Blei, who is the lead author on the original paper about topic modeling, wrote a&nbsp;<a href=\"http:\/\/www.cs.columbia.edu\/~blei\/papers\/Blei2012.pdf\">general introduction to topic modeling<\/a>. Scott Weingart has a really good post on &#8220;<a href=\"http:\/\/www.scottbot.net\/HIAL\/index.html@p=19113.html\">Topic Modeling for Humanists<\/a>,&#8221; which I think is a useful overview for anyone, regardless of the field they are in.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Credits<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This guide is a <em>very<\/em> stripped-down and adapted version of &#8220;<a href=\"https:\/\/programminghistorian.org\/lessons\/topic-modeling-and-mallet\">Getting Started with Topic Modeling and MALLET<\/a>&#8221; by Shawn Graham, Scott Weingart, and Ian Milligan, which is just one of many fabulous lessons at&nbsp;<em><a href=\"https:\/\/programminghistorian.org\/\">The Programming Historian<\/a>. <\/em>This lesson and the others at&nbsp;<em>Programming Historian<\/em> are released under a CC BY license, which I&#8217;m grateful for.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I prepared a version of this guide at first for my own use and\u00a0then deployed it in a number of workshops on topic modeling for Brown University Library&#8217;s <a href=\"https:\/\/library.brown.edu\/cds\/\">Center for Digital Scholarship<\/a>. I updated that guide in 2018 for my DigHT 315 class, making a number of improvements. Since then, I&#8217;ve continued to make small changes.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So you want to be a hero topic modeler? It turns out it&#8217;s not that difficult, thanks to the work those who created and maintain MALLET and the team behind&nbsp;The Programming Historian&#8216;s lesson on topic modeling&nbsp;(see more in credits below). The 7 Steps of Highly Effective Topic Modelers 1. Get the JDK Download and install [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"footnotes":"","_links_to":"","_links_to_target":""},"class_list":["post-370","page","type-page","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/pages\/370","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/comments?post=370"}],"version-history":[{"count":3,"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/pages\/370\/revisions"}],"predecessor-version":[{"id":424,"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/pages\/370\/revisions\/424"}],"wp:attachment":[{"href":"https:\/\/briancroxall.net\/w20dh\/wp-json\/wp\/v2\/media?parent=370"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}