Pandas .melt(): A Guide For The Perplexed

pd.melt() can be confusing because of its mediocre documentation and unintuitively-named parameters. It’s nevertheless important for getting dataframes into ‘tidy’ format for plotting purposes. 

First, we’ll discuss the id_vars and value_vars parameters. Pass into id_vars a list of columns that you DO NOT WANT CHANGED. These columns will remain as they are. Pass into value_vars a list of columns you want to be combined into a single column. pd.melt() takes the columns passed into value_vars and automatically generates a new column containing the names of those old columns. By default this column is called ‘variable‘. An additional new column ‘value‘ is generated containing the values from the old columns. 

Imagine you have a dataframe with columns 'asset‘, ‘US Dollar‘, ‘Euro‘, ‘Japanese Yen‘. The values correspond to the price of a cryptoasset like Bitcoin or Ethereum in a quote currency, like USD.

If we run df.melt(id_vars='asset', value_vars=['US Dollar', 'Euro', 'Japanese Yen']), we’ll get back a dataframe with an ‘asset’ column, a ‘variable‘ column containing ‘US Dollar‘, ‘Euro‘, ‘Japanese Yen‘, and a column ‘value‘ containing the value of each cryptoasset in each of the denominations. 

pd.melt() gives you some control over how the columns are renamed. By default pd.melt() will assume that any column not passed into id_vars is to be transformed. So instead of the above code, we could run pd.melt(id_vars='airline', var_name='asset', value_name='asset_price'), which will produce the same dataframe but with more appropriately named columns. 

Peripatesis: The Blockchain.

Doubtless all of you have heard of the cryptocurrency ‘bitcoin’, but this is merely one application built atop a deeper technology called ‘the blockchain’. Recently I have been doing some serious research on this emerging technology and I thought I’d collect the resources in one easily accessible place.

I will continue to update this page as my knowledge grows.


What is Blockchain“: Explains blockchain as a combination of open and distributed ledgers, and miners’ essential role in validating transactions on the blockchain network.

19 Industries the Blockchain Will Disrupt“:  Everyone is familiar with the disruptive potential of bitcoin for the banking industry, but have you considered what the blockchain could mean for the internet of things, supply chain management, decentralized prediction markets, insurance, charity, energy management, voting, or online music?

Blockchain 101 – A Visual Demo“: An outstanding demonstration of how the cryptographic elements and distributed nature of the blockchain make it extremely difficult to hack, clarifying terms like ‘hash’ and ‘nonce’ along the way.

Blockchain: Massively Simplified“: Elucidates the concept of a distributed, immutable ledger with an extended comparison to centralized, mutable ledgers in real estate. This talk makes it clear that what the blockchain is really changing is something more fundamental than money or any other surface-level economic phenomenon: it’s changing the way we trust. (SEE ALSO: “Sapiens“).

Dubai Blockchain Strategy“: The government of Dubai is using blockchain in a number of areas, possibly paving the way for future-oriented governments to do the same in other parts of the world.

Blockchain Demystified“: Covers much of the same material as other sources, and motivates the development of blockchain technology by noting that the original white paper debuting the concept appeared just months after the 2008 financial crisis.

Blockchain is Eating Wallstreet“: Lots of interesting graphs illustrating the impact blockchain technology will have on finance, accounting, and related value-storing/value-transferring industries.

How Bitcoin Works Under The Hood“: Is a detailed breakdown of how actual transactions occur on the bitcoin blockchain, complete with discussions of how various sorts of illegal maneuvers are avoided. (Written version)

Bootstrap a Blockchain Career“: Ivan on Tech discusses technical and educational pathways into a career utilizing blockchain technology.

Blockchain For Beginners“: Another blockchain tutorial for beginner’s from blockchain expert Andreas M. Antonopoulos. Interestingly, Antonopoulos advocates investing in learning the technical skills required to work with blockchains, he discourages speculative investing in bitcoin.

Code Implementations

Ivan on Tech walks us through a Java implementation.

Daniel Van Flymen builds a blockchain with Python.

SavJee does one with javascript.


Because I’m an aspiring hyper-poly-xeno-glot I thought I’d have a bonus section which contains blockchain discussions in some of the foreign languages I’ve studied.



(Русский) Блокчейн

(Français) Blockchain

(Español) Cadena de bloques

(Deutsch) Blockchain











Machine Ethics is Still a Hard Problem

I have now read Digital Wisdom’s essay “Yes, Virginia, We *DO* Have a Deep Understanding of Morality and Ethics” twice, and I am unable to find even one place where the authors do justice to the claims they are criticizing.

With respect to selfishness they write:
“Followers of Ayn Rand (as well as most so-called “rationalists”) try to conflate the distinction between the necessary and healthy self-interest and the sociopathic selfish.”
This is simply untrue. The heroes of Atlas Shrugged work together to bring down a corrupt and parasitic system, John Galt refuses to be made an economic dictator even though doing so would allow him limitless power, and in The Fountainhead Howard Roark financially supports his friend, a sculptor, who otherwise would be homeless and starving.

Nothing — nothing — within Objectivism, Libertarianism, or anarcho-capitalism rules out cooperation. A person’s left and right hand may voluntarily work together to wield an axe, people may voluntarily work together to construct a house, and a coalition of multi-national corporations may voluntarily work together to establish a colony on the moon. Individuals uniting in the pursuit of a goal which is too large to be attempted by any of them acting alone is wonderful, so long as no one is being forced to act against their will. The fact that people are still misunderstanding this point must be attributed to outright dishonesty.

Things do not improve from here. AI researcher Steven Omohundro’s claim that without explicit instructions to do otherwise an AI system would behave in ways reminiscent of a human psychopath is rebutted with a simple question: “What happens when everyone behaves this way?” Moreover, the AI alarmists — a demimonde of which I count myself a member — “totally miss that what makes sense in micro-economics frequently does not make sense when scaled up to macro-economics (c.f. independent actions vs. cartels in the tragedy of the commons).”

I simply have no idea what the authors think they’re demonstrating by pointing this out. Are we supposed to assume that recursively self-improving AI systems of the kind described by Omohundro in his seminal “The Basic AI Drives” will only converge on subgoals which would make sense if scaled up to a full macroeconomic system? Evidently anyone who fails to see that an AI will be Kantian is a fear-mongering Luddite.

To make the moral turpitude of the “value-alignment crowd” all the more stark, we are informed that “…speaking of slavery – note that such short-sighted and unsound methods are exactly how AI alarmists are proposing to “solve” the “AI problem”.”

Again, this is just plain false. Coherent Extrapolated Volition and Value Alignment are not about slavery, they’re about trying to write computer code which, when going through billions of rewrites by an increasingly powerful recursive system still results in a goal architecture which can be safely implemented by a superintelligence.

And therein lies the rub. Given the title of the essay, what exactly does our “deep understanding of morality and ethics” consist of? Prepare yourself, because after you read the next sentence your life will never be the same:

At essence, morality is trivially simple – make it so that we can live together.”
I know, I know. Please feel free to take a moment to regain your sense of balance and clean up the blood loss that inevitably results from having such a railroad spike of thermonuclear insight driven into your brain.

In the name of all the gods Olde, New, and Forgotten, can someone please show me where in the voluminous less wrong archives anyone says that there won’t be short natural-language sentences which encapsulate human morality?

Proponents of the thesis that human values are complex and fragile are not saying that morality can’t be summarized in a way that is comprehensible to humans. They’re saying that those summaries prove inadequate when you start trying to parse them into conceptual units which are comprehensible to machines.

To see why, let’s descend from the rarefied terrain of ethics and discuss a more trivial problem: writing code which produces the Fibonacci sequence. Any bright ten year old could accomplish this task with a simple set of instructions: “start with the numbers 0 and 1. Each additional number is the sum of the two numbers that precede it. So the sequence goes 0, 1, 1, 2, 3, 5, 8…”

But pull up a command-line interface and try typing in those instructions. Computers, you see, are really rather stupid. Each and every little detail has to be accounted for when telling them which instructions to execute and in which order. Here is one python script which produces the Fibonacci sequence:

def fib(n):

    a,b = 1,1
    fib_list = []
    for i in range(n):
        a,b = b, a+b
    return fib_list

You must explicitly store the initial values in two variables or the program won’t even start. You must build some kind of iterating data structure or the program won’t do anything at all. The values have to be updated and stored one-at-a-time or the values will appear and disappear. And if you mess something up, the program might start throwing errors, or worse, it may output a number sequence that looks correct but isn’t.

And really, this isn’t even that great of an example because the code isn’t that much longer than the natural language version and the Fibonacci sequence is pretty easy to identify. The difficulties become clearer when trying to get a car to navigate city traffic, read facial expressions, or abide by the golden rule. These are all things that can be explained to a human in five minutes because humans filter the instructions through cognitive machinery which would have to be rebuilt in an AI.

Digital Wisdom ends the article by saying that detailed rebuttals of Yudkowsky and Stuart Russell as well as a design specification for ethical agents will be published in the future. Perhaps those will be better. Based on what I’ve seen so far, I’m not particularly hopeful.

The STEMpunk Project: Goals, and How to Achieve Them

My primary focus in 2016 is the STEMpunk project, so I wanted to take a moment to make it clear exactly what I mean when I said I wanted to “shore up my techie credentials“.

Before I do, though, I want to point out how important it is to engage in the sort of plan building I’ve done below when trying to accomplish something of any magnitude. Vague slogans and unarticulated ambitions might be enough to get started on a large project, but they aren’t enough to see it through to the end. Actually finishing, or even making enough progress to not feel bad about the time invested, requires careful and consistent forethought.

With that having been said, here is the broad outline:


Funds permitting I will build my next desktop computer, use my current machine as a linux box or a media center, build an understanding of networks and computer security (including possibly getting some certifications), and hopefully make an entire virtual computer from NAND gates up.

Stage ISpend two hours a day or so reading books about building computers and putting a parts list together. Run the list by techier friends and then, if money isn’t tight, order the parts.

Even if I’m not able to actually build the computer until later in the year going through the process of researching components and how they interrelate will be an invaluable learning experience.

Stage IIComplete as much of the “build a computer from first principles” course as I can, collecting parts while I do so. When all the parts come in, build the computer, or defer the building until later in the year when I have more money to spend on components.

Stage IIIBegin with a CompTIA A+ book and work through it, spending about two hours a day. Get a certification if time permits and I think it’s worth having. Repeat the process with CompTIA’s Network+ and Security+ books. 


Stage I: Swallow my pride enough to play with some kid’s toys. The tentative list includes the SparkFun Inventor’s kit, around 5 or so soldering projects similar to this one, a 200-in-1 electronics project kit (though I’m not committing to doing all 200; I will keep it up as long as I feel I’m learning new things), the SparkFUN Photon kit, the SparkFUN Car Diagnostics Kit, the SparkFUN LabVIEW kit, and the SparkFUN Raspberry PI starter kit. The kits will probably be done in this order, though I’ll make executive decisions on moving things around. There are also some good projects at that I might pilfer if this list doesn’t hold me over.

Stage IIBegin the theory stage by reading books, completing tutorials (like this, this, or this), and taking classes like “Circuits and Electronics”, “Introduction to Electronics Signals and Measurement”, “Practical Electronics”, “Advanced Circuit Techniques”, “Power Electronics”, and “Electrical Machines”. This is a paltry sampling, of course, and I will seek out more resources as time goes on.

Stage IIIMake an inventory of all the electrical devices and systems in my house. Go through them and see how much my new-found knowledge allows me to understand, cataloging the remaining gaps. Either make a plan to fill those gaps or arrange to have a contractor/electrician come to my house and spend half a day explaining it all to me.

Stage IVReach out to an electrician buddy of mine and offer to do some free work for him in exchange for a kind of fast-paced Apprenticeship Blitz lasting a couple of weeks.


Stage I: Begin with kid’s toys like the 16-project Erector set I purchased late in 2015, a model v8 Engine Kit, a model jet engine, models of Da Vinci Clocks and Da Vinci catapults, a stirling external combustion engine, interactive “how cars work” book, perhaps others.

Stage IIMove on to theory, by reading books like “Basics of Mechanical Engineering”, “Basic Machines and How They Work”, “1800 Mechanical Movements, Devices, and Appliances”, and “507 Mechanical Movements”, “How Cars Work”, and “How Machines Work”. Take some classes like “Engineering Mechanics I”, “Engineering Mechanics II”, “How and Why Machines Work”, “Internal Combustion Engines”, and maybe even “Elements of Mechanical Design”.

Stage IIIMake an inventory of all the electrical devices and systems in my house. Go through them and see how much my new-found knowledge allows me to understand, cataloging the remaining gaps. Either make a plan to fill those gaps or arrange to have a contractor/electrician come to my house and spend half a day explaining it all to me.

Stage IVProceed through a series of real-life disassemble/repair/reassamble projects of escalating complexity. I haven’t mapped this part out completely, but I was thinking of doing something along the lines of coffee maker, water pump, weed eater motor, and cheap old motorcycle.


I’m actually most excited about this, after computing. The plan here is to use what I’ve learned in electronics, computing, mechanics, and programming to do some basic home automation. I have this vision of myself walking through my living room and casually throwing out commands in a few different foreign languages to my refrigerator, the blender, little robot arms holding up the six computer monitors my custom-built desktop is outfitted with, etc.

At a minimum, maybe I can get the refrigerator door to open on command or something.

Stage II need to keep this part lean because stage three will involve a heavy software component with a steep learning curve. I’m thinking just a few projects to get oriented and then going from there: Electronictechcrafts 14-in-1 solar robot kit, Monoprice Robot Kit, the SparkFUN RedBot kit.  

Stage IITake some classes like “Introduction to Robotics”, “Lego Robotics”, “Mechatronics”, and “Design of Electromechanical Robotic Systems”. Read “Robot Building for Beginners”. Sparkfun has got some tutorials as well.

Stage IIII’m not sure yet. I have ideas for basic home automation but no way of calibrating the difficulty of doing these things. At the end of stages I and II I may be good enough to do several of these projects, or I may be barely good enough to even get started on one. I will have to get this far and then plan more deeply.


There you have it, guys. This is what I’ll be spending most of 2016 working on!

Moving Out Of The Pre-Practice Stage

A day or so ago something encouraging happened. I was working my way through a book on coding and I managed to solve an exercise in about 15 seconds, counting debugging. Now, granted, the problem was extremely easy:

Write a function that will return the number of digits in an integer.

Here is my solution:

#python 3.3
def length_digit(num):
    length = len(str(num))
    return length

If you’ve done any programming at all you’ll probably scoff at how trivial this problem is, but I’m not bragging about it’s difficulty.  Rather, I think it means I’m beginning to get to the stage where I can actually start learning how to program.

Having gotten pretty decent at a number of things in my life, I’ve had time to observe the arc of skill acquisition.  There’s a very uncomfortable stage right when you begin learning how to do something which is sufficiently distinct from learning at later stages that I’m calling it “pre-practice”.

Take chess as an analogy.  When you’re still working on remembering how the pieces move and what their names are, you’re not practicing chess.  Sure, we generally use that verb, but until you have fully internalized piece movement, you can’t think in any real sense about deeper concepts like controlling the center of the board, to say nothing of the abyss that is chess opening theory.

Or guitar.  In the weeks when you first pick up the instrument your fingers are going to hurt when you press down on the fretboard and just holding it is going to feel awkward;  you can tell it’s going to be a while before Stairway.

Up until now I’ve been pre-practicing programming.  I’m sure I’m not done with this stage yet, especially since my programming has all been in python, but I’m getting closer.  In solving the problem named above I was above to conceive a solution immediately because I’ve become familiar with a lot of basic programming concepts, implement it quickly because I’ve developed a pretty good muscle memory for typing in python, and debug it because I’ve written and read enough code that I can spot a missing semicolon pretty quickly.

Now comes the hard part…

This Is A Test

For the past month or so I’ve been spending most of my free time learning how to program.  It’s something I’ve wanted to do since college, especially after I came across this essay, but until now I’ve simply lacked the time to really buckle down and do the work.

Well no more.

I’m not sure to what extent I’ll blog my learning process; it’s actually surprisingly difficult to write about the steps taken when solving a programming problem, mainly because there are so many tiny changes made that it’s hard to remember what you did when you finally get the damn thing running.  Plus I hate stopping to write something down when I’m on a roll, and I wanted to wait until I have code worth showing.

All of this notwithstanding, I’m posting some code today.  I’m pulling the trigger because I know that I often over-think things and I want to avoid that this time.

This snippet is a solution to the infamous FizzBuzz problem, a fairly basic coding challenge which apparently stumps quite a few programming job applicants.  It took me about three minutes to write it:

# Python 3.3

for i in range (1, 101):
    if i % 5 == 0 and i % 3 == 0:
    elif i % 5 == 0:
    elif i % 3 == 0:

This isn’t anything special.  The for loop iterates over every integer from 1 to 100, meaning  i is first equal to 1, then 2, then 3, …, 100.  Keep in mind that range stops one integer shy of the upper bound, in this case 101 .  We want the output to be “fizzbuzz” if a given integer is a multiple of both 5 and 3, “buzz” if a given integer is a multiple of 5, and “fizz” if an integer is a multiple of 3.  If a number is not a multiple of 5 or 3, we just want it to output the number.

The ‘%’ operator simply means “remainder”, so i % 5 means “the remainder of i divided by 5”, and 5 % 3 would be 2.  This is handy because it gives us a way of expressing the concept “is a multiple of”; if 15 is a multiple of 5, then the remainder of 15/5 is zero.  Ergo if i % 5 == 0 (don’t forget to use double equals for equivalence), i is a multiple of five and the program should print “buzz”.

With that, I’ve accomplished my mission of putting some code out there. It isn’t much, but it’s something.  Mastery has to start somewhere, right?

And if my reader happens to be a prospective employer some months or years down the line, at least you can see I know how to solve the Fizzbuzz problem 🙂

UPDATE [5/22/2013]
While rereading this post, I discovered that I’d made a coding mistake. My original post featured two code snippets, the first of which was this one:

# Python 3.3

def fizzbuzz():
    for i in range (1, 101):
        if i % 5 == 0 and i % 3 == 0:
        if i % 5 == 0:
        if i % 3 == 0:

If you run this code, it almost works; the error is that on an integer like 5, it prints both ‘buzz’ and the number 5, which is not what I want. The reason is because I used ‘if’ in the two conditional statements:

        if i % 5 == 0:
        if i % 3 == 0:

rather than ‘elif’:

        elif i % 5 == 0:
        elif i % 3 == 0:

I also thought that there was no difference in output when this change was made, but obviously ‘elif’ is more exclusionary than ‘if’. In hindsight I should’ve been a lot more careful in double-checking my code’s output, but the point of the exercise was to learn, and that’s what I’ve done.