Classifying SCPs, Part 1: Building a web crawler

In case you want to do the exercises and quickly get to the point where you are able to tackle one, I suggest cloning the git repository and using the tags I have created precisely for this purpose. The exercise tags take the form ex-<number>, e.g. ex-1 for the first exercise. Thus, you can simply check out the commit tagged with the exercise you want to tackle and start coding. For instance, if you want to tackle the first exercise, git checkout ex-1 will get you there. After you're done, you can compare your solution with mine by issuing git diff sol-ex-1.

Note that my solution is merely a suggestion. If you have found another one that you think might be more appropriate for the problem, feel free to leave a comment or open up an issue on github.

Also, I have provided difficulty tags (beginner, intermediate and expert) for the exercises. The beginner difficulty features exercises that hopefully will help you learn programming language features; this may consist of reading about a programming construct in the python docs and changing a keyword argument in a function. Intermediate difficulty signifies that you need to read about a certain feature in a library before being able to solve them. Finally, expert level exercises will require even more reading about library features as well as use some advanced concepts that cannot be fully explained in the text (this blog post contains one expert exercise that requires you to do some research about mocking in Python).

Do not hesitate to try out intermediate or expert level exercises even if you still feel like a beginner. Even if you are not able to solve them completely, there is much to learn from them.

Let's say that we want to crawl the article text from each SCP whose numbers are between 2 and 2500. Then the first task is to write a small function accepting a number and spitting out the URL of the corresponding SCP article. Taking a look at the URLs for SCPs #1, #11, #111, and #1111, we see that the URL format is

1http://scp-wiki.net/scp-<number>

where the number is filled up with leading zeros so that it takes up at least 3 spaces. Because I like to proceed test-driven, I create two files in the folder src/data/: A file webcrawl.py for the actual source code and a file test_webcrawl.py for tests. In webcrawl.py, let us create a prototype of our function:

1def construct_url(scp_number):
2    pass

In test_webcrawl.py, we create a prototype test to get us started:

1from .webcrawl import construct_url
2
3
4def test_construct_url():
5    assert False

From the command line, issue the command pytest. As expected, pytest complains that one of our tests fails (in this case, of course, for trivial reasons):

 1    def test_construct_url():
 2>       assert False
 3E       assert False
 4
 5src/data/test_webcrawl.py:5: AssertionError
 6
 7 src/data/test_webcrawl.py ⨯                                                                                                                                                                                                                                                                                  100% ██████████
 8
 9Results (0.14s):
10       1 failed
11     - src/data/test_webcrawl.py:4 test_construct_url

Okay, this means that our setup works. Now let us put some real assertions depending on our yet-to-write functions in there:

1def test_construct_url():
2    assert "http://scp-wiki.net/scp-001" == construct_url(1)
3    assert "http://scp-wiki.net/scp-011" == construct_url(11)
4    assert "http://scp-wiki.net/scp-111" == construct_url(111)
5    assert "http://scp-wiki.net/scp-1111" == construct_url(1111)

This time, pytest complains because our function does not do what we expect it to do yet:

 1    def test_construct_url():
 2>       assert "http://scp-wiki.net/scp-001" == construct_url(1)
 3E       AssertionError: assert 'http://scp-wiki.net/scp-001' == None
 4E        +  where None = construct_url(1)
 5
 6src/data/test_webcrawl.py:5: AssertionError
 7
 8 src/data/test_webcrawl.py ⨯                                                                                                                                                                                                                                                                                  100% ██████████
 9
10Results (0.09s):
11       1 failed
12         - src/data/test_webcrawl.py:4 test_construct_url

In test-driven development, this means we are in phase "RED" now: We have written a test that tells us exactly when we have established our desired functionality. Our target is to get to phase "GREEN" as quickly as possible. That means we can finally write some code. To fill up a given integer with zeros to at most three spaces, we can use elementary python String formatting:

1BASE_URL = "http://scp-wiki.net/"
2SCP_ROUTE_TEMPLATE = "scp-{number:03d}"
3
4
5def construct_url(scp_number):
6    return BASE_URL + SCP_ROUTE_TEMPLATE.format(number=scp_number)

Running pytest afterwards tells us that our one test has passed. We are in phase "GREEN" now. We can now safely refactor our code until we are satisfied with it. Whenever we make changes and let the tests run, we can be confident that our code still works as expected. Sometimes, this is called the "REFACTOR" phase of TDD. I will leave this phase to you in the following exercises.

After having constructed the URL, the logical next step would be to use it and request the data from the server. Fortunately, the requests library solves this issue for us. A simple call to requests.get will do. Even so, we do not need every information that a call to requests.get returns (we do not need header data from the response, we do not need the html header data…). Thus, our task will be to use the BeautifulSoup library to filter everything within the div element with the id "page-content". To test if we obtain the correct data, let us first write a main function that will serve as the entry point to our script.

 1import argparse
 2import requests
 3
 4# construct_url as before...
 5
 6def crawl_for(scp_number):
 7    url = construct_url(scp_number)
 8    response = requests.get(url)
 9    content = response.text
10    return content
11
12if __name__ == "__main__":
13    parser = argparse.ArgumentParser()
14    parser.add_argument(
15        "--number",
16        type=int,
17        dest="scp_number",
18        default=2,
19        help="Number of the SCP article to obtain.",
20    )
21    args = parser.parse_args()
22    print(crawl_for(args.scp_number))

A call to the get function of the requests library returns a Response object whose content can be retrieved via the text attribute. If the script is called via the command line, we use the argparse module to parse command line arguments. In this case, we accept an optional argument --number that defaults to 2. If you call the webcrawl.py from the commandline now, the whole HTML from SCP #2 should be printed out. However, as mentioned in the introduction, we are only interested in the children of a certain div element.

To go on in a test-driven manner, we wrap a prototype function around the response.text and write a test for it.

1def filter_for_page_content(page):
2    pass
3
4
5def crawl_for(scp_number):
6    url = construct_url(scp_number)
7    response = requests.get(url)
8    content = filter_for_page_content(response.text)
9    return content

 1from .webcrawl import construct_url, obtain_raw_content
 2
 3
 4TEST_PAGE = """
 5    <html>
 6    <head>
 7    <title>Some scp</title>
 8    </head>
 9    <body>
10    <div id="page-content">
11        <p>Some paragraph.</p>
12    </div></body></html>
13    """
14
15# test_construct_url omitted...
16
17def test_filter_for_page_content():
18    expected = """
19<div id="page-content">
20<p>Some paragraph.</p>
21</div>
22    """.strip()
23    actual = str(filter_for_page_content(TEST_PAGE))
24    assert expected == actual

Of course, this is a only basic test but sufficient for our purposes. More concretely, we want to make sure that the function extracts precisely the content from the HTML that we care about, namely the div element with the id page-content and its children. Because we have not written any code yet, pytest should signal that we are in phase RED again. Now BeautifulSoup enters the picture. The main entry point to web crawling is the BeautifulSoup object of the bs4 module. Its constructor takes the raw content. The resulting instance has a find method that can be used to find the first element with a certain name and attributes – and that's precisely the functionality we need! The function implementation comes down to this:

1# Other imports...
2from bs4 import BeautifulSoup
3
4# construct_url omitted...
5
6def filter_for_page_content(page):
7    return BeautifulSoup(page).find(name="div", attrs={"id": "page-content"})

Our tests should pass again. However, pytest gives us a few warnings that we will deal with in the exercises.

After having completed two straightforward tasks, let us come to the heart of our problem. We have extracted the main part of an SCP article and want to split it into the object class of the underlying SCP and the article text. Before we think about a solution to this problem, let us implement a prototype function.

1def split_into_label_and_text(raw_text):
2    pass

Then, let us write a test. Because this might not be straightforward, let me spell out my thoughts here. The typical input of the split_into_label_and_text function is a BeautifulSoup object containing all children of the div element with the id page-content. In particular, this BeautifulSoup object might contain a div element containing an image and it contains a div element containing the footer with the links to the previous and the next SCP article. What I want the function to do is the following:

1. It should return a tuple. The first element should be the label (i.e. the object class), the others should be the p elements containing the Object number, the Special Containment Procedures, the description, and, if present, any addenda.
2. The label should be in upper case.

3. The image and the footer should not be returned by the function.

   Having worked out these requirements, a simple test case is not hard to pull off. We can use the typical SCP HTML structure that we have worked out in the detailed introspection as a template and boil it down a little. Here's what I came up with.

    1from bs4 import BeautifulSoup
    2# The other imports and function remain untouched...
    3
    4def test_split():
    5 test_content = BeautifulSoup(
    6     """
    7     <div class="image-content">
    8         <p>Some caption</p>
    9     </div>
   10     <p><strong>Item #:</strong> SCP-xxx</p>
   11     <p><strong>Object Class:</strong> Safe</p>
   12     <p><strong>Special Containment Procedures:</strong> ...</p>
   13     <p><strong>Description:</strong> ...</p>
   14     <p>Other...</p>
   15     <div class="footer">
   16         <p>Links to other SCPs...</p>
   17     </div>
   18     """,
   19     features="html.parser",
   20 )
   21 actual_label, actual_content = split_into_label_and_text(test_content)
   22 expected_label = "SAFE"
   23 expected_content = [
   24     "<p><strong>Item #:</strong> SCP-xxx</p>",
   25     "<p><strong>Special Containment Procedures:</strong> ...</p>",
   26     "<p><strong>Description:</strong> ...</p>",
   27     "<p>Other...</p>",
   28 ]
   29 assert expected_label == actual_label
   30 assert expected_content == [str(p) for p in actual_content]

   Note that the test_content contains both a div element containing an image and another div element mocking footer data. As you can see in the list expected_content, I do not want these to be returned by the function. As is expected, this test will fail, simply because the returned None value cannot be split into an actual_label and an actual_content.

   Unfortunately, BeautifulSoup cannot help us directly to implement this function because the object class is inside a p element without any distinguishing properties. The only safe way to obtain it is to search the text for the first occurrence of the string "Object Class". Here's my implementation.

   1def split_into_label_and_text(raw_text):
   2 paragraphs = raw_text.find_all("p")
   3 obj_class_p = next(p for p in paragraphs if "Object Class" in p.get_text())
   4 paragraphs.remove(obj_class_p)
   5 label = obj_class_p.contents[-1].strip().upper()
   6 return label, paragraphs

   A lot is happening in those five lines so let me guide you through them step by step.

   1. We use the find_all method to obtain a list of all p elements.
   2. The expression p for p in paragraphs is a generator expression that lazily gives us the elements of the paragraphs list that satisfy the condition if "Object Class" in p.get_text(). The built-in function next() evaluates the generator once and thusly gives us the first p element containing the string "Object Class".
   3. We remove the p element containing the object class from the list.
   4. Finally, to obtain the label transformed to uppercase, we use the contents attribute that is a list of the children of the BeautifulSoup object to obtain the last element (index -1). Because the string "Object Class" itself is contained in a strong element, this will give us the label. The strip and upper methods are built-in methods of the string class.
   5. We return a tuple.

   This implementation still lets the tests fail. The reason is that we return all p elements as the second tuple element, even the mocked image caption and the footer data. The solution is to only look for the direct children that are p elements. This will be implemented in the exercises.

After we have obtained the label and the text of the article we have to merely persist this data so that it can be analyzed and transformed later. The easiest way is to write each article to a text file, where the first line would be the label.

Additionally, we will add a new click.argument to our command line script that allows us to submit a file path where the articles should be saved. If you have not done the refactoring exercises yet the following code samples will contain spoilers.

Here's how it could go.

 1@click.command()
 2@click.argument("filepath", type=click.Path(exists=True))
 3@click.option("--number", default=2, help="Number of the SCP article to obtain.")
 4def crawl_for(number, filepath):
 5    url = construct_url(number)
 6    response = requests.get(url)
 7    content = filter_for_page_content(response.text)
 8    label, paragraphs = split_into_label_and_text(content)
 9    with open(os.path.join(filepath, f"scp-{number:03d}"), "w") as f:
10        f.write(label + "\n")
11        for paragraph in paragraphs:
12            f.write(paragraph.get_text() + "\n")

Now everything falls into place. If you are in the root directory of the project repository, a simple python src/data/webcrawl.py data/raw/ will write the contents of the article about SCP-002 into the text file data/raw/scp-002.txt. Because we do not want to type in this command a thousand times, it remains to refactor the crawl_for function to accept a range of numbers of SCP articles whose contents should be crawled.

If you let the web crawler run with default arguments (I chose to crawl every SCP between 2 and 1000) the script will fail miserably after a few seconds.

 1~/Data Science Projects/scp [tags/sol-ex-9] λ python src/data/webcrawl.py data/raw/
 2Traceback (most recent call last):
 3  File "src/data/webcrawl.py", line 47, in <module>
 4    crawl_for()
 5  File "/Users/paul/anaconda/lib/python3.7/site-packages/click/core.py", line 764, in __call__
 6    return self.main(*args, **kwargs)
 7  File "/Users/paul/anaconda/lib/python3.7/site-packages/click/core.py", line 717, in main
 8    rv = self.invoke(ctx)
 9  File "/Users/paul/anaconda/lib/python3.7/site-packages/click/core.py", line 956, in invoke
10    return ctx.invoke(self.callback, **ctx.params)
11  File "/Users/paul/anaconda/lib/python3.7/site-packages/click/core.py", line 555, in invoke
12    return callback(*args, **kwargs)
13  File "src/data/webcrawl.py", line 41, in crawl_for
14    label, paragraphs = split_into_label_and_text(content)
15  File "src/data/webcrawl.py", line 20, in split_into_label_and_text
16    obj_class_p = next(p for p in paragraphs if "Object Class" in p.get_text())
17StopIteration

Apparently, there is a problem with finding the Object Class but it is not immediately apparent what that problem could be. Looking into the folder data/raw, we see that only seven SCP articles have been written to text files. Thus, the problem occurred when crawling the eighth article. Taking a look at the article page of SCP-008, we see that it is behind a javascript wall; the article itself is only generated when a certain link is followed. Thus, our program stops working because it cannot find the p element with the Object Class in it.

This is an example of a problem that occurs in programming all the time. You have a certain expectation and build your program with that expectation in mind; however, your expectation differs ever-so-slightly from reality and all of a sudden your program stops working. Tests will help you make sure that your program works according to your expectations but they won't help you when your expectations are off. Additionally, analyzing why your program stopped working can become a huge hassle.

Fortunately, there is a general tool that will help you make your program much more communicative and your analysis much easier: Logging.

Test for the expected, log for the unexpected.

Logging not only helps you in cases where something is wrong. It may also help you make the user experience a little better. From the very moment your script starts the user does not get any feedback until an exception is thrown. However, your user might be interested in what SCP article the program is currently dealing with because they might want to know how long it takes until it is finished.

 1import logging
 2
 3
 4logger = logging.getLogger(__name__)
 5logger.setLevel(logging.DEBUG)
 6std_formatter = logging.Formatter(
 7    "[%(asctime)s][%(levelname)-5s][%(threadName)s] - %(message)s"
 8)
 9warn_file_handler = logging.FileHandler("warnings.log")
10warn_file_handler.setFormatter(std_formatter)
11warn_file_handler.setLevel(logging.WARN)
12debug_file_handler = logging.FileHandler("debug.log")
13debug_file_handler.setFormatter(std_formatter)
14debug_file_handler.setLevel(logging.DEBUG)
15console_handler = logging.StreamHandler()
16console_handler.setFormatter(std_formatter)
17console_handler.setLevel(logging.INFO)
18logger.addHandler(console_handler)
19logger.addHandler(warn_file_handler)
20logger.addHandler(debug_file_handler)

 1# click decorators omitted...
 2def crawl_for(lower, upper, filepath):
 3    logger.debug(
 4        "Called with lower = %s, upper = %s, filepath = %s", lower, upper, filepath
 5    )
 6    for number in range(lower, upper):
 7        logger.info("Crawling number %d", number)
 8        url = construct_url(number)
 9        logger.debug("URL: %s", url)
10        response = requests.get(url)
11        logger.debug("Response: %s", response.text)
12        content = filter_for_page_content(response.text)
13        logger.debug("Content: %s", content)
14        label, paragraphs = split_into_label_and_text(content)
15        logger.info("Identified label %s", label)
16        logger.debug("Paragraphs: %s", paragraphs)
17        if label not in ("SAFE", "EUCLID", "KETER"):
18            logger.warn("Unknown label %s for number %d", label, number)
19        with open(os.path.join(filepath, f"scp-{number:03d}.txt"), "w") as f:
20            write_to(f, label, paragraphs)
21
22
23if __name__ == "__main__":
24    logger.info("Start webcrawling...")
25    crawl_for()
26    logger.info("End webcrawling...")

1[2019-11-13 15:03:19,796][INFO ][MainThread] - Crawling number 7
2[2019-11-13 15:03:20,398][INFO ][MainThread] - Identified label EUCLID
3[2019-11-13 15:03:20,398][INFO ][MainThread] - Crawling number 8
4Traceback (most recent call last):
5  File "src/data/webcrawl.py", line 78, in <module>
6    crawl_for()
7(Rest of traceback omitted)

1# Outer for loop ommitted...
2        logger.debug("Content: %s", content)
3        try:
4            label, paragraphs = split_into_label_and_text(content)
5        except Exception:
6            logger.exception("Exception when splitting for number %d", number)
7            continue
8        logger.info("Identified label %s", label)

We could let the script run now and it would fetch the first one thousand SCP articles for us; however, it will take some time. On my machine, each crawl for a single SCP article takes about 600 ms. That is to say, a thousand crawls will take about 600 seconds which is 10 minutes. Analysing the timestamps in the debug.log, it seems that most time is spent waiting for the GET request to deliver the data.

1for x in it:
2    do_something(x)

1with ThreadPoolExecutor(max_workers=64) as executor:
2    do_something(x)

1with ThreadPoolExecutor(max_workers=64) as executor:
2    executor.map(do_something, it)

 1# other imports unchanged...
 2from concurrent.futures import ThreadPoolExecutor
 3
 4# other functions unchanged ...
 5def crawl(filepath, number):
 6    logger.info("Crawling number %d", number)
 7    url = construct_url(number)
 8    logger.debug("URL: %s", url)
 9    response = requests.get(url)
10    logger.debug("Response: %s", response.text)
11    content = filter_for_page_content(response.text)
12    logger.debug("Content: %s", content)
13    try:
14        label, paragraphs = split_into_label_and_text(content)
15    except Exception:
16        logger.exception("Exception when splitting for number %d", number)
17        return
18    logger.info("Identified label %s", label)
19    logger.debug("Paragraphs: %s", paragraphs)
20    if label not in ("SAFE", "EUCLID", "KETER"):
21        logger.warn("Unknown label %s for number %d", label, number)
22    with open(os.path.join(filepath, f"scp-{number:03d}.txt"), "w") as f:
23        write_to(f, label, paragraphs)
24
25
26# click decorators ommitted...
27def crawl_for(lower, upper, filepath):
28    logger.debug(
29        "Called with lower = %s, upper = %s, filepath = %s", lower, upper, filepath
30    )
31    with ThreadPoolExecutor(max_workers=64) as executor:
32        executor.map(
33            crawl, (filepath for _ in range(lower, upper)), range(lower, upper)
34        )

After having run the web crawler, watching the 64 threads delightfully crunching through the pages and punching them into text files, it is time to take a look at the results. There are quite a few warnings and errors logged into our warnings.log. Let's take a look at them and see if we have to modify the web crawler and re-run it once more.

1grep "\[ERROR\]" warnings.log | wc -l

1grep "Unknown label" warning.log | wc -l

 1    def test_split_with_leading_colon(self):
 2        test_content = BeautifulSoup(
 3            """
 4            <div class="image-content">
 5                <p>Some caption</p>
 6            </div>
 7            <p><strong>Item #:</strong> SCP-xxx</p>
 8            <p><strong>Object Class</strong>: Keter</p>
 9            <p><strong>Special Containment Procedures:</strong> ...</p>
10            <p><strong>Description:</strong> ...</p>
11            <p>Other...</p>
12            <div class="footer">
13                <p>Links to other SCPs...</p>
14            </div>
15            """,
16            features="html.parser",
17        )
18        actual_label, actual_content = split_into_text_and_label(test_content)
19        expected_label = "KETER"
20        expected_content = [
21            "<p><strong>Item #:</strong> SCP-xxx</p>",
22            "<p><strong>Special Containment Procedures:</strong> ...</p>",
23            "<p><strong>Description:</strong> ...</p>",
24            "<p>Other...</p>",
25        ]
26        self.assertEqual(expected_label, actual_label)
27        self.assertEqual(expected_content, [str(p) for p in actual_content])

1AssertionError: 'KETER' != ': KETER'
2- KETER
3+ : KETER

1label = obj_class_p.contents[-1].strip().upper().replace(": ", "")