After many hours of perl programming, my fears are realized: I find myself wanting to do things in python as I might do them in perl…
if ((my $count = scalar(@$rows)) > 50)
{
# do some stuff...
}
It’s just so convenient…
Thanks to this stackoverflow question, I’ve learned a useful git trick:
git update-index —assume-unchanged [
…]
and its counterpart:
git update-index —no-assume-unchanged [
…]
This pair of commands lets me keep a default file in the repo (e.g. a sample config) and modify it locally without git complaining about changed files. Handy!
I encountered a mysterious bug in Flask-Markdown: footnotes in one document were polluting others. I noticed footnotes being duplicated in a markdown-formatted sidebar, and then I found them showing up on new pages altogether.
The (approximate) cause was obvious: markdown was keeping state between different renders.
It turns out that Python-Markdown allows you to instantiate a renderer with options set and reuse that, if you choose, and Flask-Markdown takes advantage of this. However, when using Python-Markdown in that way, you need to reset() the instance between renders.
Solution: change line 69 of Flask-Markdown’s markdown.py from:
return Markup(self._instance.convert(stream))
to
return Markup(self._instance.reset().convert(stream))
Simple.
Unfortunately, it looks like the maintainer of Flask-Markdown hasn’t been active on github in quite a while, so this problem may persist.
All of my public projects have been suffering, lately (not that I’ve ever been the most constant of writers), because I’ve been working in the background on a tool to aid me in my research.
Some background: Until now, I’ve been collecting my copious notes in wikidPad, a very nice personal wiki application. If all you need is a text-based notebook with some simple formatting and organizational capabilities, it’s great. I recommend it to everyone. My needs are a bit more extreme, though. I’ve currently got notes on about eighteen hundred individual items, along with pages of organizational and planning information. Making use of all the data I have stored in wikidPad was getting to be impossible, and, worse still, some of my notes were scattered in different locations elsewhere due to my needing features not supported by wikidPad.
I’ve tested the various available research notebook tools (Docear, Zotero, Mendeley, ReadCube, Evernote, OneNote, …) but most are heavily oriented toward research artifacts being PDFs (does no one use photographs, or recorded interviews, or anything? just PDFs?), and none had remotely powerful enough metadata and reporting capabilities.
So, I finally bit the bullet and created my own tool. It’s ugly and user-unfriendly right now, but for my purposes it’s already more useful than wikidPad, and I continue adding features as needs arise. The benefit of using custom-built software is that I can readily make any required modifications. If I want to see all of my screenshots of character selection screens for games released between 1987 and 1996, ordered by the name of the developer, with games I’ve never reviewed highlighted in green, I can have that in a couple of minutes. Television episodes about gambling written by people born before 1960? People who starred in movies from the nineties based on games released in the eighties? Games I haven’t beaten developed by defunct companies? Any desired query can be constructed in a few minutes, as long as the information is there. And a lack of information was never my problem.
I don’t know if anyone else would find this useful. The lack of tools that did even remotely what I want seems to indicate that my needs are a bit idiosyncratic. I’ll probably release the code once I’ve got it in a less embarrassing state, all the same.
BeautifulSoup is really excellent. I should have started using it ages ago. In probably less than an hour, I turned my custom-built horrible, fragile, regex monstrosity of a parser into a much more robust and, more importantly, understandable scraper with half as many lines of code.
The lesson, as it often is, is to use the right tool for the job.
Status update.
| Project | Status | Notes |
|---|---|---|
| Books to sort | 3140 | Rapid progress lately. |
| LT imported | 1689 | No progress lately. |
| TMFT | 73 | About a third done. |
Recently, I saw a new game from Zachtronics, TIS-100, which was released on Steam as an early access title on the first of June. In some ways, calling it a game is overstating it: it’s basically just a collection of programming problems with an interface. The catch is that you’re programming in an assembly language on a virtual machine with unusual architecture. Problems beyond the simplest will generally require you to take advantage of parallelism (which is the primary distinguishing feature of the VM) to solve, which leads to some rather different solutions for traditional problems. It’s neat, and I suggest checking it out.
Obviously, a game like that has a rather limited target audience. Case in point: I have previously created my own little VM with a fake assembly language to play with. The game is clearly made just for me, but how many others are likely to be similarly interested? About 11,000 so far, apparently.
I had good fun playing the game, which reminded me that I’ve been meaning to get better at assembly. I’ve made some (very) simple programs in x86 assembly before, but I could really use some more practice and study. I looked into learning a little Z80 assembly for the Game Boy, which dovetails nicely with my interest in video games, and even built a little test ROM, but ultimately it seemed more useful to study something I’m a bit more likely to use. So, I’ve been reading Programming from the Ground Up, which teaches Linux x86 assembly. It’s a little old, and could seriously use some proofreading, but it’s a decent resource.
One error to note here: in Chapter 4, on page 63, there is a lovely diagram indicating the current state of the stack at a certain point during the execution of the code. Lovely, but wrong. It has the order of the “Base Number” and “Power” reversed. When I first saw it, the reversed order made me think that the top of the stack was at the top of the diagram, when in fact it is not. I worked it out, of course, but it did cause me to do some double- and triple-checking of the code to be sure. Caveat lector.
As for what purpose I’ll eventually put this to… I’ve got some ambitions to write an emulator, and I hope to transfer this knowledge to other platforms. In particular, I’m interested in looking at C64 assembly. It’d be nice to look at some of those old games with a better idea of what’s going on! For now, though it’s just learning for the sake of learning. I’m a way off from doing anything very interesting with it.
The last few days, I’ve been looking into writing apps for android. I found a framework, Kivy, that would let me write in python, which seemed like it’d be nice.
Results are mixed.
After some hours, I have indeed successfully created an android app. However, the process was painful and tedious and I remembered partway through how much I hate writing user interfaces, as I discovered that practically all of the coding time was spent on the UI. My lack of familiarity with the framework led to some truly awful code, which I’ve slowly been refactoring. A thousand lines it should not take for a simple database app.
On the plus side, the app does seem to be useful and serving its purpose, which is more than I can say for a lot of what I code, so… I’d call this a minor success.
After about six hours of work, I’ve completed an initial release of a new project: cgrr-gamecube. At the moment, it is able to parse GameCube GCI file headers, plus decode the banner from the GCI file (if it’s in CI8 format).
It also supports writing the GCI files back out, but only modifications to the header are supported. That means no editing the banner, since it’s stored in the save data blocks, and not the header. In the future, I intend to add functions specifically for replacing the banner.
Decoding the banner was something of a pain. The CI8 image format isn’t very complicated, I suppose. It’s a 16bpp paletted format with 5 bits per channel plus one transparency bit, with the image stored as a series of 8x4 pixel tiles (documentation forthcoming—I used this page, myself). Not complicated, but annoying to work with, since I had to first rewrite the colors to a more usable format, and second reorder the image data so it wasn’t all tiles. Though in retrospect I suppose I could have actually decoded the image as tiles and then pasted the tiles into a new image in the right positions. Not sure which would have been better.
Anyway, everything went pretty smoothly, my dislike of the image format aside.
I think I may put together a frontend so I can extract GCI files rapidly, to support my (potential) future efforts in decoding GameCube save files. One more item for the todo list, I suppose.
After many months of procrastination, I’ve finally split up my cgrr project and transferred it to github. It took quite a while, but in the end I think it will be much more maintainable and useful with each of the parser modules taken separately than it would be with them all jammed together pretending to be plugins for some yet-unwritten master program. If I want to write some wrapper around them all later, they’ll perfectly well support it, but until then, all the plugin stuff is just an extra mess—YAGNI.