Building a Python-powered website

One of my core programming projects is a Flask-app that can do security-related lookups of URLs, domains, IP addresses and email addresses. It is one of the few projects I actually took some time for to (minimally) document on this website (see the project here). So far it has been able to look up generic information like the IP address of a website, when a domain was registered and where, its SPF (Sender Policy Framework) and DMARC (Domain-based Message Authentication, Reporting and Conformance)-records to indicate email security and authenticity. It is also able to look up more specific security information like what VirusTotal and AbuseIPDB think of the domain or IP address. All of this information is displayed in (somewhat) dynamic tables that provide basic highlighting (grey undetected, green harmless, red malicious).

Learn Python The Hard Way (really)

The idea came after starting a final assignment of the book Learn Python 3 The Hard Way by Zed A. Shaw. The assignment was to make a Flask app that would run a text-based adventure game. While I liked the challenge of the project and wanted to get more into Python by learning the Flask framework, I noticed I wasn't motivated by the assignment and the already challenging experience I had with the book. By the way, that is not a negative take on the book, more a description of where I was at the time, learning (and seriously struggling with) Python: I felt like I wasn't really in control of what I was doing and a project like this was more intimidating than inspiring.

Make it useful

To smooth these bumps in the road, I started thinking how this difficult project could actually be of use to me. Meaning, could it provide some useful function for me, in my day to day work? As a security operations center (SOC) analyst, one of the things you tend to do, is look up the validity and reputation of hosts, meaning IP addresses, URLs, domain names, email addresses, etcetera. Is a domain vulnerable to phishing, has a URL already been flagged as malicious by antivirus vendors? If so, the unexpected behavior that your security tooling flagged for you, could be malicious. Of course, your tools usually already do a lot of this work for you. But it's still inspiring to see your own hand-written code do something useful. So, I set out to do just that.

A few moments later...

More like a few months later! In between my day job and self-study to become a better security analyst (malware analysis, security engineering, cloud security), I found enough time to get the Flask app running on a local server and make it useful for myself. But, at a certain point, I got stuck: the site did what it needed to do, but it couldn't handle for example multiple inputs (copy and paste a number of domains and look each up sequentially). Also the file upload (upload a text file with hosts in it) function needed more work. Finally, I was only doing all of this on my local development server, not in any representative way. When I started documenting the project, this got me thinking: shouldn't I take the next step and turn this into an actual website?

Saved by a forum post

I understood that the next step this project needed to take, was to make it real: move away from the development server and run the code on an actual domain! But...how? This website and its subdomains all run on the same self-hosted Yunohost-server, which takes a lot of the administrative work away from me (thankfully). But the (very rich) Yunohost ecosystem did not seem to have a ready-made Flask-app solution ready. Then I stumbled (again) onto the 'My Webapp'-project (link to repo). With this application, you can install a custom web app, complete with a database, proxy configuration ready to go, SFTP access, that you can customize to build your own web application, basically. But how could I leverage this into hosting a Flask-app? Luckily, I wasn't the only one looking for the answer to this question. This detailed post on the Yunohost-forum made it look quite simple. And lucky for me: it was! Now I have a (absolutely barebones Hello World) Flask-app ready on the subdomain sec.joostagterhoek.nl. Hooray! 🎉 Now, looking through the Flask documentation and considering the use cases and risks of my web app, I know what to do...

Work to do

What I have to do (and have to learn) is the following:

• Figure out rate limiting for my web app (because I am using my own (free) API keys for services like VirusTotal and AbuseIPDB)
• Turn the code I have now into a wheel (following the Flask documentation on deployment)
• Figure out a save way for environment and secrets handling (API keys for example)
• Make the website look a lot better (it's all text and a few colors right now 😅)

I know that these tasks will keep me quite busy for the next few weeks or months. But! It's great to have a shiny new goal to work towards 🚀. See you soon! 👋