Developer Drain Brain

November 14, 2015

Manna from heaven

Filed under: Uncategorized — rcomian @ 5:51 pm

I’ve been a developer using the Microsoft stack (C++ with MFC, C#) since 1998. I’ve never professionally used any other stack for a significant length of time. Don’t get me wrong, I want to. I *really* want to.

I’m not a Microsoft (or apple) fan, and the recent ASP.NET releases have helped to codify why.

We’re waiting for the next version of ASP.NET. It’s going to be the greatest version ever, it’s pretty much copying node.js in its architecture, we’re getting cross platform support, large parts are being open sourced and it’s going to be amazing.

Only it isn’t. I’ve been watching a colleague use the current beta of vnext. Basically, it includes bower and the gulp build chain which works fantastically well – neither of these are microsoft technologies. Everything else has been painful. Worst of all, SignalR has been removed from the betas and is now unavailable in the stack. It won’t be back for the main release either.

Apparently, Microsoft think that a fully fledged, top of the range webstack doesn’t have to include push services.

That’s bad enough, but it’s the reason that this is bad that really gets me. I realised all us C# developers are sitting here waiting for “manna from heaven”, for the goods to just fall out of the sky and we’ll lap them up whatever they are. We can’t credibly start a new project on the old version of ASP.NET, knowing that the new version is right around the corner and that it’s what everyone is going to use regardless of whether that’s the right thing or not, we’re just going to use it because it’s what we’re told we should use. It’s what works with C# and .Net, the framework we’re paid to develop with.

I’m comparing this with *everything else* that’s out there. There have been many languages and environments for making websites: php, python, ruby. They’ve all come about organically from people who’ve needed something and taken the effort to create it. Node.js is the most recent manifestation of this.

I *like* Node.js, although I was sceptical of it for a very long time. But it works. And the way it works is interesting. I’m not choosing any particular framework when I choose node. I can choose express, or sails, or hapi or any of the others. Or none, I can do it all raw if I want to.

The “next” version of node is exciting, but when things start going wrong or getting stale, someone forks it and brings it back up to speed. This happened. It now merged back into a proper platform. No “manna from heaven”, more “this isn’t working, let’s do it ourselves”.

If the next version of express isn’t any good, people will just move to another framework, or keep using the old one, someone will probably even keep maintaining an old version if people use it.

I can’t see any way that will suddenly become incompatible with the next version of express. We’re not going to lose major features like this. That said, it doesn’t matter, we could still use socks.js, or raw websockets if we wanted. There’s thousands out there.

No one is telling me I must use any particular framework. Choice is its own burden for sure, but I can choose frameworks that fit my style, attitude and the problem at hand. When any of those change, we can move frameworks without ditching the whole stack.

It’s the same with linux. When I use windows, I’m choosing one set of apps. When I choose OSX, I’m choosing another. When I choose linux – well, I’m choosing another set of apps for sure. But there’s a difference. You can’t *choose linux*, you choose *a* linux. I choose gentoo, because it works how I think. You might choose ubuntu because you like how it works.

That’s great, we’ve both chosen a system that looks and works the way we think computers should function. Notice what we haven’t chosen tho – we haven’t chosen a different set of apps. Pretty much everything that’s written for ubuntu will work on gentoo and vice versa (support may be sketchy, that’s another problem tho).

My point is tho, everywhere else in the industry, things aren’t given to us from on high. Things are developed that solve a particular problem, if they solve your problem, you’re free to choose them, or not. If you don’t you don’t have to switch out the entire stack.

I can carry on using my same linux apps if went to ubuntu. Or fedora, or mint. I can carry on using if I switch away from express – or use it on its own. I can carry on using express if I choose not use anymore.

I don’t know if I’m being clear here. I think my point is very subtle. But I know what I’m not liking. I’m not liking waiting for Microsoft to pour Manna from heaven on me, and knowing that I’ll be forced to eat it up, regardless of what I actually think of it, because I know that everyone else is going to be using it as well and the industry will demand it of me regardless. And Microsoft *will* make it “good enough”, for sure. I just wish I could choose the new ASP.NET. Or choose not to do the new ASP.NET.


September 25, 2013

Incrementally calculating Mean, Variance and Standard Deviation in T-SQL

Filed under: Uncategorized — rcomian @ 4:20 pm

So, today I had a performance issue. I found a piece of SQL that inserted a lot of data in to a table, one row at a time.
For each row, it inserted a random value into the table, then in to another table inserted the mean and standard deviation of all the values up to that point. In T-SQL the way to do this is quite straightforward:

INSERT INTO @value_table VALUES (@value)
INSERT INTO @stats_table VALUES ( SELECT SUM(val) / COUNT(val) AS mean, STDEV(val) as stdev FROM @value_table )

Of course, the problem with code like this is that calculating the statistics iterates over every existing value each time it’s called – and it’s called for every row that’s inserted.
For me, it started to creak after just a few hundred thousand rows – and my machine got very warm.

The way to solve this sort of problem is to calculate the stats incrementally – that is save the current state in variables and just add the next number in each time rather than going and recalculating the whole lot each time.

Incrementally calculating the mean is quite straightforward, we keep a @sum variable to which we add the @value and a @count variable which we increment by 1 each time.

This gives us something like the following:

INSERT INTO @value_table VALUES (@value)
SET @sum += @value
SET @count += 1
INSERT INTO @stats_table VALUES ( SELECT @sum / @count AS mean, STDEV(val) as stdev FROM @value_table )

That deals with the mean, so how to we calculate the standard deviation like this?
Well, first of all, I had to remind myself what the standard deviation was. Thanks to Standard Deviation and Variance page on Maths is Fun, I found that the standard deviation is just the square root of the variance.
Which is a good start. It means we can calculate the variance incrementally and then get the square root of that value as the variance.
But what’s a variance? High-school maths was a long time ago, but it is, of course, the sum of the square of differences between the value and the mean.
Well we know the mean for each point, but things are a little tricky since the mean changes for every data point we add in – and we need to calculate the difference between the each datapoint and the last mean value, which implies we need to go back to revisit each data point with the new mean each time and recalculate everything.

But there must be some hope, the new mean changes from the old mean in a measurable way, we must be able to take advantage of that to recalculate everything.

Fortunately, it turns out someone who actually knows math has already worked this out, thanks to the Maths Stack Exchange site, someone had already asked and answered that question.

Now, when I went to that link, I got a little scared. I really wasn’t sure what all those numbers actually represented, but if you read it carefully, you can parse it out.

But how do you go about coding this and checking that your calculation is correct?

Well, it took a little fiddling, but I finally managed to pull something together, so I give you – incrementally calculating the Mean and Standard Deviation in T-SQL and checking that the values are correct by also calculating it longhand:

DECLARE @generating TABLE (val REAL)

@vals_Count INT = 0,
@vals_Sum REAL = 0,
@vals_Mean REAL = 0,
@vals_Variance REAL = 0,
@vals_LastMean REAL = 0,
@vals_LastVariance REAL = 0,
@nextVal REAL = 0

— Define the values that we’re going to insert here
DECLARE @toinsert TABLE(id int identity(1,1), w REAL)
INSERT INTO @toinsert (w)

SELECT @i = 0, @max = max(id) FROM @toinsert

WHILE @i < @max

SET @i += 1
SELECT @nextVal = w FROM @toinsert WHERE id = @i

— Add this value to our table
INSERT INTO @generating VALUES(@nextVal)

— Save the last mean and variance values
SET @vals_LastMean = @vals_Mean
SET @vals_LastVariance = @vals_Variance

— Incrementally calculate the new mean
SET @vals_Count += 1
SET @vals_Sum += @nextVal
SET @vals_Mean = @vals_Sum / @vals_Count

— Incrementally calculate the new variance (if you do this when count = 1 you get a divide by zero error, probably because the concept is meaningless)
IF (@vals_Count > 1)
SET @vals_Variance = (((@vals_Count-2) * @vals_LastVariance) + ((@vals_Count-1) * POWER((@vals_LastMean – @vals_Mean), 2)) + (POWER(@nextVal – @vals_Mean, 2))) / (@vals_Count-1)
SELECT @vals_Variance = 0

— Select the
SELECT @vals_Count as id, @nextVal as inserted, SUM(val) / COUNT(val) as mean, STDEV(val) as stdev, @vals_Mean as ‘incMean’, SQRT(@vals_Variance) as incStdev FROM @generating


August 12, 2013

Oliver Hotham – It’s Great When You’re Straight… Yeah

Filed under: Uncategorized — rcomian @ 1:32 pm

This is a mirror of a mirror.

Reposted from:
More of the story here:

< Begin >
Oliver Hotham posted the following, and is being told to take it down by Straight Pride UK. Below is the full text from his blog. ETA: This text is from a document entitled ‘Press Release.’

AUGUST 3, 2013

There has never been a better time to be gay in this country. LGBTI people will soon enjoy full marriage equality,public acceptance of homosexuality is at an all time high, and generally a consensus has developed that it’s really not that big of a deal what consenting adults do in the privacy of their bedrooms. The debate on Gay Marriage in the House of Commons was marred by a few old reactionaries, true, but generally it’s become accepted that full rights for LGBTI people is inevitable and desirable. Thank God.

But some are deeply troubled by this unfaltering march toward common decency, and they call themselves the Straight Pride movement.

Determined to raise awareness of the “heterosexual part of our society”, Straight Pride believe that a militant gay lobby has hijacked the debate on sexuality in this country, and encourage their members, among other things, to “come out” as straight, posting on their Facebook page that:

“Coming out as Straight or heterosexual in todays politically correct world is an extremely challenging experience. It is often distressing and evokes emotions of fear, relief, pride and embarrassment.”

I asked them some questions.

First of all, what prompted you to set up Straight Pride UK?

Straight Pride is a small group of heterosexual individuals who joined together after seeing the rights of people who have opposing views to homosexuality trampled over and, quite frankly, oppressed.

With the current political situation in the United Kingdom with Gay Marriage passing, everyone is being forced to accept homosexuals, and other chosen lifestyles and behaviours, no matter their opposing views. Straight Pride has seen people sued, and businesses affected, all because the homosexual community do not like people having a view or opinion that differs from theirs.

Are your beliefs linked to religion? How many of you derive your views from scripture?

Straight Pride aims are neutral and we do not follow religion, but we do support people who are oppressed for being religious. Only today, Straight Pride see that two homosexual parents are planning to sue the Church because they ‘cannot get what they want’. This is aggressive behaviour and this is the reason why people have strong objections to homosexuals.

You say that one of your goals is “to raise awareness of the heterosexual part of society”. Why do you feel this is necessary?

The Straight Pride mission is to make sure that the default setting for humanity is not forgotten and that heterosexuals are allowed to have a voice and speak out against being oppressed because of the politically correct Government.

Straight Pride feel need to raise awareness of heterosexuality, family values, morals, and traditional lifestyles and relationships.

Your website states that “Homosexuals have more rights than others”. What rights specifically do LGBTI people have that straight people are denied?

Homosexuals do currently have more rights than heterosexuals, their rights can trump those of others, religious or not. Heterosexuals cannot speak out against homosexuals, but homosexuals are free to call people bigots who don’t agree with homosexuality, heterosexuals, religious or not, cannot refuse to serve or accommodate homosexuals, if they do, they face being sued, this has already happened.

Straight Pride believe anyone should be able to refuse service and speak out against something they do not like or support.

There is a hotel in the south of England, called Hamilton Hall which only accepts homosexuals – if this is allowed, then hotels should have the choice and right to who they accommodate.

What has been the response to your campaign?

The response to Straight Pride’s formation has been as expected; hostile, threatening, and aggressive. Homosexuals do not like anyone challenging them or their behaviour.

We have had support from many people saying that if homosexuals can have a Pride March, and then equality should allow Heterosexuals to have one too. After all, the homosexual movement want everyone to have equality.

Why would you say that heterosexuality the “natural orientation”?

Heterosexuality is the default setting for the human race, this is what creates life, if everyone made the decision to be homosexual, life would stop. People are radicalised to become homosexual, it is promoted to be ‘okay’ and right by the many groups that have sprung up.

Marriage is a man and a woman, homosexuals had Civil Partnerships, which was identical to Marriage with all the same rights, they wanted to destroy Marriage and have successfully done so.

If you could pick one historical figure to be the symbol of straight pride (just as figures like Alan Turing, Judith Butler or Peter Tatchell would be for Gay Pride) which would you choose?

Straight Pride would praise Margaret Thatcher for her stance on Section 28, which meant that children were not taught about homosexuality, as this should not on the curriculum.

More recently, Straight Pride admire President Vladimir Putin of Russia for his stance and support of his country’s traditional values.

How do you react to anti-gay attacks and movements in Russia and parts of Africa?

Straight Pride support what Russia and Africa is doing, these country have morals and are listening to their majorities. These countries are not ‘anti-gay’ – that is a term always used by the Homosexual Agenda to play the victim and suppress opinions and views of those against it.

These countries have passed laws, these laws are to be respected and no other country should interfere with another country’s laws or legislation.

We have country wide events which our members attend, and ask people their opinions and views, on such event at Glastonbury this year was very positive with the majority of people we asked, replied they were happily heterosexual.

For the record, Straight Pride did not respond to these questions:

“Pride” movements such as Gay Pride and Black Pride were making the argument that the stigma against them meant that proclaiming their “pride” was an act of liberation from oppression. Can being heterosexually really compare?

A problem that Gay rights activists cite is the issue of bullying, and the effect this can have on young LGBT people. Do you think a similar problem exists with straight children being bullied by gay children?

I will obviously add to this if they do respond.

You can follow Straight Pride on Twitter here and see their Facebook page here.

April 1, 2013

Everything you need to know about Bitcoin

Filed under: Uncategorized — rcomian @ 5:21 pm

After looking in to bitcoin over the last few months I’ve decided to put together everything I’ve found in a blog post.
I’ve tried to make this post largely non-technical, more of a user guide than a technical guide. Its job is to introduce the concepts that are useful, then if you want technical detail about specific parts, they are easy it find.


Bitcoin is designed to be a currency – a form of money that you can use to pay for things. It sits alongside other currencies like the US Dollar, the Euro, the Pound Sterling and the others. And just like you can buy Euros with Dollars and vice versa, you can buy Bitcoins with Dollars – and vice versa.

Bitcoin is unlike other currencies in one crucial way – it’s not controlled by any government. In fact no-one is in control of Bitcoin. It’s managed entirely by a peer-to-peer Network which anyone can join just by running a simple program. If the Network agrees that you have Bitcoins, then you have Bitcoins. What’s clever about this is that it is done in a way which is secure even though it’s distributed. It uses very high strength cryptography in innovative ways and it’s unbreakable with today’s technologies. So it’s secure even though there are no central banks which keep track of how much everyone has.

The currency abbreviation for Bitcoin is BTC. So the currencies listed above would be USD, EUR, GBP & BTC.

Bitcoin infrastructure

Bitcoin uses numbered accounts, much like those of the famous Swiss banks. An account has a number (called an Address), a secret code (called a Private Key) and a value – the number of Bitcoins it contains.

You can have someone send Bitcoins to your account just by giving them its Address, but you need the Private Key to send those Bitcoins to someone elses account. The Private Key is super secret, you never give it to anyone – if you did, they could use it to send your Bitcoins anywhere they liked! Every Address has its own Private Key.

An Addresses is a long complex number and Private Keys are even longer. For example, an Address with a Private Key might look like this: 31uEbMgunupShBVTewXjtqbBv5MndwfXhb, 5Kb8kLf9zgWQnogidDA76MzPL6TsZZY36hWXMssSzNydYXYB9KF.

You’re not expected to type or remember these numbers. When you use Bitcoin, a program on your computer or phone remembers all these numbers for you, collecting a group of Addresses with their Private Keys into a Wallet.

Bitcoin Addresses are different from Swiss bank accounts in that you don’t need to register them with anyone. Whilst a Swiss bank might give you a number to use, you generate Bitcoin Addresses yourself. They’re basically random numbers and you can make as many as you like – millions or billions if that’s what you want – and no-one will care if you do as there’s no cost at all to anyone else.

A Transaction is when Bitcoins are moved between Addresses. The Network keeps track of every Transaction that has ever happened – and if no-one has transferred Bitcoins in to an Address, then that Address has no Bitcoins in it. So whilst you can create as many Addresses as you like, they will all be empty until someone moves some Bitcoins in to them. The Network only knows about Addresses that have had some Bitcoins sent to them.

Because Addresses are basically free, it’s normal to use Addresses just once and forget them once they’re empty. For example, if you shop online and pay with Bitcoins, it’s normal for the website to create a new Address just for you to send Bitcoins to. Once you’ve paid, that Address will never be used again. If you change your mind and don’t pay, that Address will just be forgotten.

Getting Bitcoins

So if you’re starting out with Bitcoins, how do you get them? This is the same problem as with any currency – imagine you were suddenly interested in Japanese Yen – how would get some? You either have to accept them in payment for something, or you buy them through an exchange.

At the moment it is difficult to buy Bitcoins. This is because normal online payments, such as by Visa or PayPal can be reversed by the buyer claiming not to have received the goods. Because Bitcoin transfers aren’t reversible, this means that none of the big players are willing to accept the normal online payment methods for Bitcoins (and those that have went bust because of this fraud).

One method I have found is using VirWox. If you want to move small amounts, you can deposit 50 Euros (or equivalent USD or GBP). The complication is that the only way to buy BTC is with lindon dollars (SLL), so you have to deposit in your currency, buy SLL and then buy BTC. This makes the purchase price very high, but is one of the few reliable ways to purchase Bitcoins with a standard online payment. If you have an avatar in SecondLife you can buy SLL using the SecondLife website, then transfer into VirWox and buy BTC that way – that route might allow you to move more money, at a price.

How much are Bitcoins worth?

This is a tricky question, Bitcoins have no intrinsic value – they don’t represent an amount of gold or the promise of some work done. They’re also not linked to any other currency, so their price doesn’t go up or down with any other currency. Bitcoins are worth exactly what people will pay for them. Fortunately there is a thriving market for Bitcoins and their value is measurable. At the time of writing you could buy 1 Bitcoin for about 62 GBP, 73 EUR or 94 USD. However, that value changes a lot, quickly – just one month ago that price would have been about 20 GBP. The Bitcoin market is relatively small and unknown which makes it volatile, as time goes by it should become more stable, but at the moment it’s extremely unpredictable.

Spending partial Bitcoins

A single Bitcoin is quite a large currency unit. If a Bitcoin is worth 94 USD, it’s pretty much a hundred dollar bill. If the minimum we could give anyone was a hundred dollar bill it would make buying small things like newspapers very difficult. Fortunately, a Bitcoin is just a number associated with an Address. This means that it doesn’t have to be a whole number, we could have 0.5 Bitcoins, or even 0.001 Bitcoins. Currently, Bitcoins are allowed to be split down to 8 decimal places. That means the smallest amount we can transfer is 0.00000001 Bitcoins – which is worth way less than a penny in any currency at the moment.

Incidentally, 0.00000001 Bitcoins is called a Satoshi, after the person who originally invented the system.

Anonymity and tracking Bitcoins

Remember that an Address is just a random number and doesn’t need be registered with any central authority. This means that knowing an Address doesn’t tell you anything about who owns that Address. In fact, there’s nowhere to register your name and personal details within Bitcoin even if you wanted to, it just doesn’t exist. This makes Bitcoin anonymous, very much like paper money. There is something to be aware of, however, in that every single transaction ever made is recorded publicly. This means that if someone can trace an Address to you specifically, then they can just lookup every Transaction and see where all the Bitcoins for that Address came from and went to. There is no way to hide that a Bitcoin came from this Address and went to that Address. Transactions are public knowledge and there’s no way to fudge the history after the fact – it’s not just a ledger, changing the historic records would make all the current records invalid and would be detected and fixed immediately by the Network – this is all part of how Bitcoin keeps its integrity.

In order to maintain as much privacy as possible, the recommendation is to never use an Address more than once if you can avoid it. Most clients have features to help with this, for example, if you have an Address with 5 Bitcoins in it, and you send 2 Bitcoins to someone, the client will create a third Address, called a Change Address and move the remaining 3 Bitcoins to it as part of the Transaction. This way, no-one knows if it was 2 or 3 Bitcoins that got transferred to someone else. If you then pay someone else 1 Bitcoin, it’s coming from that new Address, so no pattern can be built up.

Sometimes you have to re-use Addresses. Donations are one example: some people have a line in their email or forum signature that says “Donations welcome here: (Address)”. Anyone can see how much has been donated that way and now there’s a link from that Address to that person. It’s not the end of the world, it’s just something to be aware of.

Good privacy hygeine helps the whole Network. If everyone’s private, then it very hard for anything to be tracked. If privacy is ignored in some quarters, then it reduces the privacy for everyone as there becomes known points where Bitcoins can be traced through. We’re not just talking about privacy from the government here, we’re talking about privacy from anyone: russian mafia, people traffickers, abusive spouses, stalkers – they all have the same access to the public record as anyone else and are possibly more interested in tracking where certain Bitcoins go than any investigator. Keeping your own privacy helps everyone.

The mechanics of sending Bitcoins

Bitcoins are sent from Address to Address in Transactions. A Transaction has 2 sides, From and To and you can have any number of Addresses on each side. After using Bitcoin for a while, you might find that you have a dozen Addresses, each with some small number of Bitcoins. An example transaction is here:

1Hq53US8QKFnWiUzbDKFWmb4rsGf5Zkacb: 10
17ahwGCn396bVv8WsBKXvGMJUZ9DeiAHa2: 0.02

1KhbSE6ehE7dP6jHrgaGsssVkXkGHLS6Gq: 1.02
1CYq1y7KFbWdDZVMSyy7RkUQhwEurT1hvc: 9

In this case, you can see 10 Bitcoins have been moved out of the first address and 0.02 Bitcoins moved out of the second Address. Normally this would leave the first 2 Addresses empty, but that doesn’t have to be the case.
Then from those Bitcoins, 1.02 Bitcoins are moved into the 3rd Address and 9 Bitcoins moved in to the last Address. See how those Addresses tell you nothing about who owns them? Also, was this a payment for 9 Bitcoins, or 1.02 Bitcoins? Does the person paying have 9 Bitcoins remaining in their wallet, or 1.02? From this transaction, it’s impossible to tell.

So what happens when you decide to send someone some Bitcoins. First, you need an Address to send to. This is straightforward, they should tell you as part of any transaction. Next you normally just tell your Bitcoin client, which is a program running on your computer or phone, how much you want to pay, and what address to pay to. Your client will then automatically select some Addresses with enough Bitcoins in them, create a Change Account and create a Transaction like the one shown above.

Your client will then validate that everything is correct – that you really do have those Bitcoins in those Addresses and that the Addresses you’re sending to are valid Bitcoin Addresses. Your client will then use the Private Keys that it kept with each Address and mark the transaction with proof that you are allowed to send money from that Address. It then sends the Transaction in to the Network.

Every Bitcoin client is part of the peer-to-peer Network. Usually, you will be connected to 8 or so other Bitcoin clients. When you send a Transaction, you send it to each of those 8 other clients. Those clients could be anyone, just people like you who use Bitcoin. Their clients will then validate the Transaction themselves, check that the Private Key marks are legitimate (this is a mathematical operation and doesn’t require looking anything up) and then check that the From Addresses really did have enough Bitcoins in them to do the Transaction. This takes fractions of a second, if the Clients are happy that the Transaction is legitimate, they’ll send the Transaction on to everyone connected to them, who will also validate the Transaction. So you send to 8 other Clients, they each send to 8 other Clients and so on.

In this way, your Transaction will spread across the entire Bitcoin Network in a matter of seconds. The person you’re sending Bitcoins to should see your Transaction arrive in their Client very quickly. If this is a quick, low value transaction then this might be enough to say that all is well.

Because the Transaction is validated by every Client it goes through, it means that if you hack your own Client to send out an invalid Transaction, it will simply be dropped by all the other Clients because it’s invalid. This makes the Network very resilient to hacking – you’d have to own the majority of Clients to stand a chance of your invalid Transaction being accepted. Since the Clients can run every platfrom out there and have multiple implementations, the chances of any one hacker being able to do that is very low.

But there are still some theoretical attacks that can be performed, so if you’re paranoid (and you should be just as paranoid as you are about fake bank notes) you can wait for the next phase – Confirmation.

Confirming Transactions

Confirmation happens when your Transaction gets included in to the Block Chain and your peers all agree that the Block Chain is valid.

The Block Chain is the public record of every Transaction. Special clients, called Miners, put every Transaction in to the Block Chain. Once a Transaction is broadcast in to the Network, it normally takes about 10 Minutes or so to start getting Confirmations that the Transaction is valid, but it can happen more quickly or much more slowly depending on how busy the Network is, how much you paid in Fees and luck.

Fees and Getting your Transaction Confirmed Quickly

It’s worth noting that it’s entirely up to the Miners which Transactions they include in the Block Chain. Since the Network is peer-to-peer, there’s no guarantee that every Miner will see every Transaction, so you can’t say the Block Chain is invalid just because it doesn’t contain a certain Transaction. It’s perfectly valid for parts of the Block Chain to be completely empty – and that does happen.

So why should a Miner spend their valuable computing resources to include your Transaction in to the Block Chain? One thing you could do is sweeten the deal. Every Transaction includes a special field – a Fee. When a Miner puts your Transaction in to the Block Chain and gets it accepted by the Network, they’re allowed to collect the Fees on the Transactions they added. This is one way that Miners get to make money, by collecting all these Fees. For a normal Transaction, an average Fee is between 0.0005 and 0.001 Bitcoins, but if your Transaction includes a large number of Addresses, the Fee should be higher since the Miner has to do more work validating each Address. Your Client software will suggest a Fee. You can pay as much or as little as you want, including nothing at all. However, if you don’t include a Fee, or a Fee that’s below the market rate, then it can take a very long time before anyone gets around to including your Transaction in to the Block Chain. Your Transaction will normally be included within a couple of days, but there are no guarantees. At the moment, in practice, not including a Fee can delay Confirmation by a few hours. As Bitcoin grows in popularity, that could get much worse. Normally, Transactions get included in to the Block Chain in 10 Minutes or so.

How the Block Chain Works

Bitcoin keeps a public record of every Transaction that ever occurred in it. It does this by collecting a bunch of Transactions together and combining them in to a Block. At any moment in time, several thousand special Clients, or Miners are competing to create the next Block. When you send your Transaction out in to the Network, these Miners will grab your Transaction and include it in the next Block they’re trying to make (if it’s valid). Once someone successfully makes a Block, it’s transmitted to all the Clients just like your Transaction was. Just like your Transaction, each Client confirms that all the parts of the Block are valid and that all the Transactions are valid and if so, pass the Block on to their peers. Each Block is connected to the Block before it and forms a chain of Blocks going back in history and containing every Transaction that ever occurred. This Block Chain forms the public record.

Anyone can run a Miner. The Miners all compete to make the next Block and it’s designed to be very hard to do so, so you need very powerful computers to be an effective Miner. The difficulty is tuned so that on average, only one block is made in the entire Network every 10 minutes. This is why getting your Transaction Confirmed takes about 10 minutes or so.

Mining is a difficult process and you don’t need to be a Miner to use Bitcoin. Bitcoin does, however, rely on Miners to create its Block Chain and has an interesting methods to pay Miners for their work.

Bitcoin relies on its Block Chain. There is only one Block Chain: one list of blocks in sequence. Each Block contains information from the previous Block, so if anyone tries to change a Block back in the Chain, even if the change itself is valid, it means that the following Blocks would no-longer be valid (since they would contain information for the original Block) – to make a change to history you would have to re-create every Block from the one you change to the present day. This is generally considered difficult enough that it’s impossible. More, Clients themselves contain the signature of a recent Block in their source code. So if someone did try to recreate the entire history, the Clients wouldn’t accept any changes from before their built-in Block anyway.

Creating a single Chain using multiple distributed Miners is a difficult process, but mechanisms are built in to the Network to ensure that it goes smoothly. First, it’s made so difficult to produce a Block that it’s unlikely that two valid Blocks will be produced at the same time. Secondly, mechanisms are built in so that if two or more competing valid Blocks are produced, the network votes on one which becomes the official Block and the other one is forgotten.

There are several websites that let you examine the Block Chain for yourself. One popular site is

Why run a Bitcoin Miner?

If Mining is so crucial to the Network, yet so difficult, how do we make sure that people can be bothered running the Mining software? After all, it takes effort to set it up, it also ties up your computer and it takes electricity. Yet some people spend thousands of dollars buying custom built machines just to run Miners on. Is this all out of the goodness of their hearts?

To some extent, yes, one reason run a Miner is simply because you like the idea of Bitcoin and want to support it. But you can also do that just by running a client that forms part of the Network does the validation.

The best way to ensure that people do things is to pay them, and Bitcoin has 2 mechanisms for this. One we’ve already seen is to collect the Fees from each transaction. This will be a continuous income and will never expire.

The other way is to take a bounty. Whenever a Miner creates a Block, the first Transaction they put in is a free credit to the Address of their choosing. This is the only kind of Transaction in the system that doesn’t move Bitcoins from one place to another – this is the source of brand new Bitcoins. All the clients are coded to agree that the first Transaction should be this free credit, so it’s considered a valid Block. Since every Miner will be trying to credit their own Address, it means that no two Miners will ever produce exactly the same Block at exactly the same time.

The amount that a Miner is allowed credit themselves goes down over time. For the first 4 years, it was 50 Bitcoins per Block. This halves every 4 years (every client knows this) and at the moment, the bounty is for 25 Bitcoins per Block. This puts an upper limit on the total number of Bitcoins that will ever be in the system. Something like 21 million Bitcoins.

Once all the Bitcoins have been mined, no more will ever be produced. At that point, it’s expected that people will be Mining for the Fees only. It’s quite clever, by reducing the bounty gradually over time, it weens Miners off of expecting the bounty and on to expecting the Fees. If Bitcoin becomes phenomenally successful, the Fees alone should be quite valuable (looking at the Blocks as I write this, the fees alone are often worth between $15 & $30).

You should be able to see why people bother investing in building Mining machines. At 25 Bitcoins a block, winning a block is worth around 2500 USD at todays prices. The general expectation of the value of Bitcoins is that they’ll go up and it’s definitely true that the difficulty of Mining Bitcoins will go up, so some people think it’s worth Mining whatever they can right now.

Ensuring security for Transactions

How exactly do we ensure that a request to move Bitcoins from one Address to another is legitimate? It’s an extremely important question. And it’s impressive to learn that no-one has ever stolen Bitcoins by breaking the system. People have had Bitcoins stolen, but that was by breaking in to computers and stealing the Private Keys, something that is just as commonplace when, for example, people steal credit card numbers through keyloggers, etc.

So Bitcoin is considered secure, but how?

It all hinges on Public Key Cryptography. If you don’t know anything about Public Key Cryptography and Hashes, this section won’t make much sense. However, it’s a critical part of modern computer security, so I suggest you learn at least the basics on how it works.

Bitcoin uses Elliptic Curve Cryptography (ECC), which has not had any known attacks against it and can use much smaller key sizes compared to DSA and RSA. ECC routinely uses 256bit private keys and are considered secure. The key size is important since all Transactions are stored forever, so using a key a quarter of the size will require much less storage over time. ECC may also be faster than RSA or DSA for the same cryptographic strength, although this is less clear.

The Public and Private Keys are generated together using the standard algorithms for ECC. The Address is essentially hash of the Public Key, constructed by using a couple of different hashes and combining them. The result is that you can validate an Address easily and it’s incredibly unlikely to mistype an Address and end up with another valid Address. Since the Address is a hash of the Public Key, you can always tell if a Public Key matches the given Address by deriving the Address from the Public Key again and checking that they match.

ECC allows you to sign a message with the Private Key and validate that signature with the corresponding Public Key. So when a Transaction is created by your Client, it signs the Transaction with the Private Key and adds the Public Key so that anyone can validate the signature. When we receive a valid Transaction, this is what we know:

  • We know the Address, but cannot go from the Address to either the Public Key or Private Key.
  • We know the Public Key, we can validate that the Address matches the Public Key, but we cannot get the Private Key from the Public Key.
  • We know the Transaction has been signed with the Private Key. We can validate that signature with the Public Key and stronly infer that the Private Key known by the person who created the Transaction.

From this, we can tell without looking at any registries whether the person who made the Transaction has the Private Keys that correspond to the Addresses that the Bitcoins are being moved from. We can tell from the Block Chain whether those Address had enough Bitcoins in them at that point in time and we can tell that the Addresses Bitcoins are being sent to are valid and haven’t been mistyped.

The other side of the guarantee is to ensure that no more Bitcoins come out of an Address than go in. This called double spending. In everyday usage, we do this by using physical tokens – we hand over a 10 dollar note to the store clerk, for example. We can’t then go to the next store and hand over the same 10dollar note. Online, our bank keeps a tight track on how much money we have in our account and every transaction reduces the amount available immediately.

In Bitcoin, the Block Chain is the golden standard, but there’s a gap between when we submit the Transaction and it gets incorporated into the Block Chain. This means that it’s conceivable for someone to send a Transaction into one side of the Network and another Transaction into the other side at the same time. Eventually, both Transaction will spread throughout the whole Network, but it could take a few seconds, even some minutes. Consider what happens when a Miner tries to add these Transactions to a Block.

First, the Miner may only have one of the Transactions and this could make it immediately into a Block, if luck is working that way. Then, when any Miner receives the next Transaction, they notice that there isn’t enough Bitcoins in the Address and simply reject it. The Transaction will never be made in to the Block Chain. If any Client or Miner receives both Transactions before either make it in to a Block, then it’s ultimately up to them whether they accept one of the Transactions and drop the other, or drop both.

This double-spend attack is only a problem if you’ve accepted a payment that ultimately doesn’t make it in to the Block Chain. However, it’s extremely difficult to do this in any meaningful way, if you receive a payment notification and it isn’t quickly invalidated, it’s likely to be good. In practice the first transaction is likely to make it in to the Block Chain, if you receive the second Transaction you’re client is likely to have already received the first Transaction and can tell that there aren’t enough Bitcoins available.

However, the only way to be really secure is to ensure that the Tansaction has made it in to the Block Chain and that it has been accepted by the rest of the Network. You Client will let you know when that has happened because the Transaction will either be marked as Unconfirmed, or having some number of Confirmations.

Making Blocks hard to create

One of the last pieces of genius in the Bitcoin Network is how Blocks are made. More particularly, how the Network is tuned so that Blocks are hard to make. This is an important problem, if hundreds of Blocks were made every second, sorting out the order of the Blocks would become an impossible problem. By slowing this down, we can choose one from a small list, even if the final choice is somewhat arbitrary there will be very few conflicts where this is a problem.

So what’s the secret? It’s that each Block has to be hashed to match a specific value. What each Miner does is create a Block header which includes the hash of the previous Block, the hash of the Transactions in the current block and an arbitrary value called a Nonce. This header is then hashed and this becomes the hash used in the next Block’s header. The trick is that the hash is only valid if it conforms to a certain pattern. Specifically, it must start with some number of 0’s. If the header doesn’t hash to that value, the Miner increments the Nonce and tries again. It keeps doing this until it finds a Nonce that causes the header to hash to a valid pattern.

If we consider that the hash is a SHA256 – so there are 256 bits we can play with. If we require just the first bit to be 0, then 1 in every 2 Nonces will create a valid hash. If we require 2 zero bits, then that goes down to 1 in every 4 Nonces. If we require 10 zero bits, then we would have to try on average 1000 Nonces before we find a match.

We can keep requiring more bits until the Miners must hash the header a very large number of times before it finds a valid block. As more Miners join and more powerful machines are created, the number of zero bits required increases, making it more and more difficult to find a valid Block. Every Client can work out how many zero bits are required just by looking at the Block Chain, so if a Miner tries to cheat, the Block is simply rejected by the other Clients in the Network without any central authority saying that it’s bad. The difficulty is tuned so that the whole Network only produces a valid Block every 10 minutes.

As an example, the valid Nonce for a recent Block as I was writing this was: 3,544,225,952 – so it likely took 3.5 billion attempts to find a valid Nonce for that Block. I say likely, because there’s no requirement that a Miner should try Nonces sequentially, it could just try random numbers. But it should be clear that the system can make it very difficult if it needs to. This is why being a Bitcoin Miner needs a lot of computing power – you need to be able to create a lot of hashes in order to stand a chance of finding a Block.

So if every Miner is just incrementing a value until they find a valid Block, why don’t the fastest computers all find the valid Nonce at the same time? And how can anyone except the fastest computers compete?

First of all, not all Miners will include the same Transactions in the same order. Second, each Block contains a unique Transaction – the free credit that every Miner can give themselves. Since each Miner credits their own account, each Block is by definition unique. This means that each Miner is looking for a different Nonce to make their Block valid. This means that if you’re lucky, even the slowest computer could find a valid Block. In general it’s statistical, you will win Blocks at the ratio of your computer’s hash rate / the Network’s hash rate. This makes it a race to create more powerful hardware for Miners, which is overall a good thing for Bitcoin as it means that its very hard to hijack the Block Chain.

Pooling Mining resources

If you look at the hashrates of the Network, things might looks bleak. For example, maybe the Network can do 60 trillion hashes a second. If your machine can do 500 million hashes a second you’d find a valid Block once every 2.5 years. However, at current Bitcoin rates you’d still be earning on average £2 a day. If only there was some way to earn that £2 a day without having to wait (on average) 2.5 years for each payout …

Fortunately, there is a solution. If you join a Mining Pool, you can combine your computing power with hundreds of others, all trying to find the correct Nonce for the same Block. If someone in the Pool finds a valid Block, the Block earnings are distributed to everyone who contributed at a pro-rata rate. The exact details of the payout differ pool-by-pool, but in general you earn between 50% & 100% as much as you would have earned by Mining on your own, but instead of getting a large payout every few years, you get a small payout every few days. How often you get paid depends on the combined hashrate of everyone in the Pool. Most Pools are large enough that they’ll win a Block at least every few weeks. Some are large enough to win several Blocks a day.

Create a free website or blog at