MinorFs2

People who have read my blog, have read my article, of been at any of my public talks know about the problems of the unix $HOME and $TEMP facilities. MinorFs, a set of least authority file-systems aimed to solve this problem in a relatively ‘pure’ way. That is, it was a single granularity, single paradigm solution that gave pseudo persistent processes their own private storage that could be decomposed and delegated to other pseudo persistent processes.

A few years after I released MinorFs and a few years after I wrote a Linux Journal article about MinorFs, although its painful to admit, its time to come to the conclusion that that my attempts to make the world see the advantages of the purity of model that the stack AppArmor/MinorFs/E provided has failed.

At the same time, the problem with a $HOME and $TEMP directory that are shared between all the programs a user may run is becoming a bigger and bigger problem.  On one side we have real value being stored in the users $HOME dir by programs like bitcoin. On the other side we have HTML5 that relies more and more on the browser being able to reliably store  essential parts and information of rich internet application.

The realization of these two facts made me come to an important conclusion: Its time for major changes to MinorFs.  Now I had two options. Do I patch a bunch of changes on top of the existing Perl code base, or do I start from scratch. In the past I had tried to get MinorFs accepted as an AppArmor package in Ubuntu. At that point I ran into the problem that MinorFs had rather exotic perl modules as dependencies.  So if I ever want a new version of MinorFs to be accepted as a companion package for AppArmor, I would have to rewrite it quite a bit to not use those exotic dependencies. Add to this the major changes needed to get MinorFs as practical as it gets without compromising security, I had to come to the conclusion that there was little to no benefit in re-using the old MinorFs code.  This made me have to change my earlier assertion to: Its time to write a new version of MinorFs from scratch.

So what should this new version of MinorFs do in order to make it more practical? What should I do to help ang det it packaged in the major distributions that package AppArmor? The second question is easy, be carefull with dependencies. The first question however turned out to be less simple.

I have a very persistent tendency to strive for purity of model and purity of design. But after a few years of seeing that such purity can lead to failure to adopt, I had to take a major leap and convince myself that where purity gets in the way of likelihood to be adopted, purity had to make way.

After a lot of thinking I managed to concentrate all of the impurity into a single place. A place that allowed for configuration by the user in such a way that a purity sensitive user, packagers and administrators could create a relatively pure system by way of the config, while practically inclined users, packagers and administrators could just ignore purity al together. The place where I concentrated the impurity is the persistence id service. This service that didn’t exist in the old MinorFs maps process-id’s to persistence-id’s, but it does this in a way where one process might get a persistence-id that implies a whole different level of granularity than the persistence-id an other process maps to. Where the old MinorFs had only one level of granularity (the pseudo persistent process), MinorFs2 allows different processes to exist at different granularity levels according to the needs and possibilities of their code-base.

This is the base of the first of my practical approaches. It suffers one program that requires the existing user level granularity to co-exist with for example an other program that benefits from living at the finest granularity level that MinorFs2 provides.  I tried to come up with every potentially usable granularity level I could come up with. In practice some of these levels might turn out to be useless or unneeded, but from a practical viewpoint its better to have to much than to have the one missing that would be the best fit for a particular program.

So what would be the most important practical implication of allowing multiple granularities down to user level granularity? The big goal would be : Allow to simply and effectively  replace the usage of the normal $HOME and $TEMP with the usage of MinorFs2.

We should make it possible to mount MinorFs2 filesystems at /tmp and /home and have all software function normally, but without having to worry about malware or hackers with access to the same user id gaining access to their secrets, or being able to compromise their integrity.

This practical goal completely ignores the benefits of decomposition and delegation, but it does make your computer a much safer place, while in theory still allowing application developers an upgrade path to fine grained least authority.

An other practical choice I had to make was replacing the use of symbolic links with  using overlay file-systems for minorfs2_home_fs and minorfs2_temp_fs and dissalow  ’raw’ access to minorfs2_cap_fs to unconfined processes.  I won’t get into the details about what this entails, but basicaly I had to abandon the golden rule that states: ‘don’t prohibit what you cant enforce‘.  Unconfined processes have access to the guts of the processes running under the same uid. This makes them capable of stealing the sparse capabilities that MinorFs uses. I took the practical approach to:

• Limit the use of raw sparse caps to delegation and attenuation (less to steal)
• Disallow unconfined processes from directly using sparse caps

This is an other practical issue that makes stuff a bit impure. In the pure world there would be no unconfined processes in a pure world, no way to steal sparse caps from the guts of an other process. So what do we do, we break a golden rule and close the gap as good as we can. Knowing that:

• If the unconfined malicious process can convince a confined process to proxy for it shall be able tu use the stolen sparse cap.
• If a non malicious unconfined process wants to use a sparse cap it can’t.

It hurts having to make such impure design decisions,  it feels like I’m doing something bad,  badly plugging a hole by breaking legitimate use cases.  I hope that the pain of  the practical approach will work out being worth  it  and I’ll be able to create something with a much higher adoption rate than the old MinorFs.

Json made easy

In my last blog ‘An ode to the cast operator’ I talked about using a combination of 3 simple C++ syntactic sugar constructs in order to make a library API simple to use:

• Cast operators
• Subscript operators
• Value semantic smart pointer wrappers.

I’ve now spent some more hours with the problem library that made me write that last blog and have come to a useful wrapper library that I would like to share with you.

I’ve called this library ‘JSON made easy for C++’ or JsonMe++ for short, and its now available on github. The result isn’t a full bidirectional JSON lib, at least noy at this moment. It just provides the facility for accesing the data inside a JSON document or string, the functionality I myself needed. Given that this library, that is a wrapper for the Glib json library is now complete up to a point that it should be usable to anyone, I wanted to revisit this subject to show how the resulting API is indeed trivial in use by the merit of the above 3 flavours of syntactic sugar.

So lets look at a piece of code using this library:

jsonme::JsonMeLib jsonlib;

try {

   jsonme::Node topnode=jsonlib.parseFile(pathToFile);

} catch (jsonme::ParseError &e) {

  ..

}

This simple piece of code is actualy the hardest part of the code. If the file path provided is invalid, or the file isn’t valid JSON,

an exception will get thrown. But lets not look at that. The interesting part is in the fact that there is no ‘new’ or ‘*’ anywhere in the code.

The user of the library doesn’t need to juggle with pointers, wrap them in smart pointers in order to take care of resource management,

mix pointer semantics and value semantics in the code, etc.

The library takes care of all of this, and all the user is exposed to are JsonMeLib and Node objects that can be used using value semantics,

internally taking care of effective resource management and efficient and relatively light object copy behavior. This is where we see

value semantic smart pointer wrappers in action. Both JsonMeLib and Node are value semantic smart pointer wrappers. So when the result of

parseFile is assigned to topnode, under the hood smart pointers are taking care of proper resource management, without the need of the user of the library being exposed to resource management. A subject that for many C++ developers is a difficult subject and large source of subtle bugs. The library user actually needs no knowledge about RAII, smart pointers or resource management in C++ in order to use the library without fear of creating resource management bugs, That is, if there are resource management bugs it would be the fault of the library author.

JSON knows basicaly 3 types of node’s:

• Objects
• Arrays
• Scalars

The first two is where the second piece of syntactic sugar comes in: the subscript operator. The library uses size_t indexed subscript operators for arrays

and std::string indexed subscript operators for objects.

  size_t index=0;

  jsonme::Node firsfoobar=topnode["foolist"][index]["bar"];

The overloaded subscript operators together with our value semantics smartpointer wrapper give the JSON object an interface that is friendly to the user of the library in that its intuitive and does not require the library user to get into the library type internals.

Combining these two with cast operators,  we make it the library user even easier:

   long long magicvalue=topnode["magic"];

   std::string  owner= topnode["foolist"][index]["bar"];

The library takes a step back from this to allow for validation. Node has a nodetype() method that can return  jsonme::INVALID. Next to this between Node and the primitive scalar types the library has a class Scalar that the user may use when working with JSON data structure not hard coded into the C++ code.  That is, the library user can choose to look into some more of the API for finer control, but the bottem line is that the provided syntactic sugar makes the amount of API internals a pretty small surface to work with.

An ode to the cast operator

Working on a project that is written 90% in python that uses JSON for its configuration, I had to parse the python generated JSON in C++. Not a problem, C++ boost has an excellent and usable library that has JSON support, the Boost Property Tree library. Unfortunately however the target platform comes with the 1.40 version of the boost library, and guess what, the boost property tree library became first available in the 1.41 version. Off to look for a different JSON library that is available as a standard package for the target platform. After searching the pachaging system of my target platform I found it comes with JSON-Glib.

First I started out being happy about the fact that I found an alternative JSON library for my project, but my joy quickly resided when I realized it was a C++ library with one of those API’s. Untill we get the rout node of our JSON document all looks fine, but once we have our first JsonNode while expecting to get a transparent API that maps effortlessly to the base C++ types, we end up being exposed to all kinds of JSON-Glib classes, and some classes from the core Gnome library.

Having written some of   those libraries myself, I know its easy to start believing in the merits of your own project internal type system, thinking nothing about exposing it to your library users.There are however a few useful tricks you can use with relatively little effort to decouple the knowledge of your library internal type system from the need of your library users to be exposed to that knowledge.

• Cast operators
• Subscript operators
• Value semantic smart-pointer wrappers.

When designing an API for a library using these 3 tools, it is possible to greatly decrease the learning curve for using your library. You provide syntactic sugar that hides much of your internal type system from the part of your user base that isn’t that interested in learning about your no-doubt brilliantly designed library type hierarchy.

The cast operator

The most interesting of the 3 tools is the C++ cast operator. Adding a cast operator to a type allows to simply assign an object class that can validly be represented as a base type, for example an std::string, to a variable of that type. Yes, its just syntactic sugar, but its syntactic sugar that allows your library user to not learn about the internals of your library. We shall come back to the cast operator later, but lets start off by an example of an API class that uses cast operators. The following is part of a wrapper library for JSON-Glib that I am currently working on:

typedef enum {FLOAT,INTEGER,STRING,BOOL,NULLVAL} scalartype;

class AbstractScalar {

public:

virtual ~AbstractScalar(){}

virtual jsonme::scalartype type()=0;

virtual operator long double()=0;

virtual operator long long()=0;

virtual operator std::string()=0;

virtual operator bool()=0;

virtual bool isNull()=0;  };

 }

Subscript operator

An other piece of powerful syntactic sugar that is provided by languages like C++ and Python that support operator overloading is the subscript operator. For our JSON-Glib wrapper for example the subscript operator could be used both for accessing JSON object members by key, or for accessing JSON  aray members by index.  Again an excerpt from the wrapper library I’m working on:

typedef enum {OBJECT,ARRAY,SCALAR} nodetype;

class Node;

class AbstractNode {

public:

virtual ~AbstractNode(){}

virtual jsonme::nodetype type()=0;

virtual Node operator[](std::string name)=0;

virtual size_t size()=0;

virtual Node operator[](size_t index)=0;

virtual operator Scalar()=0;

};

If we combine this with the previously discussed cast operator, our JSON wrapper library could in theory be used as follows:

std::string firsthostname= rootnode[0]["host"]["name"][0];

As should be clear from this, the user should potentialy have verry litle knowledge if any about the types the API is composed with.

Value semantic smart-pointer wrappers.

A third piece of syntactic sugar that may not be directly obvious is the use of value semantic smart-pointer wrappers in our API.  As a rule, a library should not burden the users of that library with the issue of having to think about resource management. C++ provides the concept of smart pointers to help with correctly managing resources. Exposing and forcing the use of smart pointers can be a proper choice, for example returning an auto_ptr or a unique_ptr from a factory is much better than returning a raw pointer. In the library wrapper we are discussing here however, it may be much more suitable to completely hide the usage of smart pointers (that have pointer semantics) by providing a value semantics wrapper for the smart pointer.

class Node: public AbstractNode {

boost::shared_ptr<AbstractNode> mNode;

public:

Node(AbstractNode *node):mNode(node){}

~Node(){ delete mNode();}

jsonme::nodetype type(){ return mNode->type();}

Node operator[](std::string name) { return (*mNode)[name];}

size_t size() { return mNode->size();}

Node operator[](size_t index) { return (*mNode)[index];}

operator Scalar() { return (*mNode);}

};

conclusion

I hope that with the above examples I’ve shown how a adding a little bit of syntactic sugar to your library API can make a lot of sense when you want to spare your library users from a relatively steap learning curve. Although I have myself often been to lazy and presumptuous with respect to my own libraries and their type hierarchy to be thorough and sufficiently frequent in the usage of these constructs in my API’s, but running into JSON-Glib while only needing to parse a config file made me aware once more of the user perspective of those API’s. Remember that if your library is any good it will have more users than it has developers. Users that will be grateful for a reduced learning curve. And if its a library for a tiny market, than you may possibly still increase the number of potential users by adding these 3 ingredients to your API. I like all 3 of the syntactic sugar components described above, but to me the C++ cast operator is their absolute champion.

Taming mutable state for file-systems.

 

After my January 2009 Linux Journal article on MinorFs, I had a talk titled taming mutable state for file-systems that I gave several times over the past two year. Actually I gave this talk 7 times in 2009, once more in 2010, and my last appointment to give this talk this month (may 2011) bounced at the last moment.  I guess, that however much I enjoyed giving this talk, its unlikely that I will be giving it again. As a way to say goodbye to the material of this talk, I will dedicate this blog post to talking about my favored  talk While I did put the slides to this talk online, they were not completely self explanatory, so hopefully this blog post can open up my talk, that I probably won’t be giving any more times, to other people interested in least authority and high integrity system design.

As I hate phones going off in the middle of my talk,  I like to start my talks with a bit of scare tactics. I have a chrystal watter jug with a no phone zone sticker on it that I fill with water and a fake Nokia phone.  I than show my jug to the people in the room and asking them to please set their phones to silent, informing them that if they have problems doing so, than I would be happy to offer my jug as a solution to that problem.  Untill now, these scare tactics have worked and I have been able to give my talk without being interrupted by annoying phones each of the times.

My talk starts off with something I stole shamelessly from a presentation by Alan Karp. I talk to my audience about an extremely powerful program. A program that has the power to:

• Read their confidential files.
• Mail these files to the competition.
• Delete or compromise their files.
• Initiate a network tunnel, allowing their competition into their network

Then we let the audience think about what program this might be before showing a picture of solitaire, and we explain that while we don’t expect solitaire to do these things, it does have the power to do these things. As there will always be some Linux and Mac users who enjoy laughing at the perceived insecurity of Microsoft products,  I than go on explaining that this is not just a problem in the Microsoft world, but that other operating systems have exactly the same problem. That is, Linux for example is just as bad, so we change the picture in our slide from solitaire to my favorite old school Linux game sokoban.

Next we expand on the problem, saying that while sokoban might be OK, there are a lot of programs running on our system, written by even more people, with even more people in a position to compromise one of these programs into doing bad things. Then we extend it further by talking about network applications like a web browser, and how even if these are benignly written, an exploitable bug might easily transform these programs into something that will exploit the extensive powers that it is given.

Now other than Alan’s talk where I stole the solitaire stuff from, I don’t go on talking about how much power solitaire/sokoban has to do all these things, and how according to the principle of least authority solitaire/sokoban should not have the right to for example access that confidential ‘global’ data, but I take the opposite approach in that I talk about what this confidential data might be, and that it had no reason for being global in the first place.  I say that if we have an editor that was used to create a secret, this editor has no power to protect that secret from sokoban.

Than I went on to paint an extended picture of a secret where we wanted to share confidential information written in our editor with a friend using e-mail. I painted a scenario where the user would have 20 programs that she run on a regular basis on her system. 3 of these programs were our editor,  a mail client and encryption software.  I tried to explain that only the editor and the encryption software had any business with access to the secret.

Than we get to what I feel is the core of my talk. Mutable state. I have a slide that very graphically shows the potential difference between two ways of dealing with mutable state. Either as shared mutable state or as private mutable state. won’t describe the slide in detail, but it involved a rather vivid pictures of  lavatories, and what being public could lead to.

From our lavatories we came to the point that we were going to look at file systems and global mutable state, where we had to come to the conclusion that with all the users programs running as the same user, the file system, for all practical purposes, only gave us public mutable state and no private mutable state. From that we went back to look at the core problems with the concept of global mutable state, which are:

• That it can potentially be modified from anywhere.
• That any subsystem may rely on it.
• That it creates a high potential for mutual dependencies.
• That it makes composite systems harder to analyze or review.
• That it makes composite systems harder to test.
• That it basically in many cases  violates the principle of least authority.

Now with the problem so clearly identified, and with a small kid at home who loves to watch Bob the builder, I couldn’t resist but while creating my slides to let Bob ask the question ‘can we fix it?’….. Taking a few steps back to have to come to the conclusion that the problem might have already be fixed in an other domain, computer programming.

In computer programming we have different kinds of problematic shared mutable state:

• global variables
• class variables
• singletons

I like to refer to global variables as the obvious evil of the devil we know, class variables as the lesser evil, and singleton’s as the devil we don’t.  So now we show what computer programing has done to solve the problem. We show that OO has given us private member variables and a concept known as pass by reference, and that it, in its basic form has given us two lesser evils (singletons and class variables) we can use to avoid the bigger evil (global variables).  Now from two sides the lesser evils are under fire in computer programing. From the high integrity side that gives us the object capabilities model (a sub set of OO that excludes implicitly shared mutable state), and from the TDD side where dependency injection is used as a way to address the testability issues that come with implicitly shared mutable state. Now we dive into one side of this, object capabilities, and more specifically a cute little language called E. This language shows us some of the capability based security principles that we can apply on our file system problem.  Next to this, as we will later see, this language can provide us with the roof of a whole high integrity building that we will try to build.

So what makes E, or object capability languages as a whole such a great thing? Basically, not focusing primary on Trojans but on exploitable bugs, its about the size of the trusted code base. If I want to protect my secret, how much lines of code do I need to trust? In any non ocap language the answer basically is ‘all of them’. If for example an average program is 50000 lines of C or C++code, than if this program had access to my secret, I would be trusting 50k lines of code with my secret. Using an ocap language, its quite reasonable to have a core of a program designed according to the principle of least authority (POLA) that truly  needs access to the secret. The great thing about an ocap language is that it allows you to easily proof that only that for example 1000 lines of code to be considered trusted. So using an ocap language for trusted programs,  we could reduce the size of the trusted code base per trusted program for our secret to a few percent of its original size.

Now starting at the roof with building is seldom a good idea, we have Bob the Builder start out with a look at the foundation. The foundation we choose consists of two components:

1. The AppArmor access control framework for Suse and Ubuntu.
2. FUSE : Filesystems in userspace, a Linux/BSD library+kernel module for building custom filesystems in userspace.

AppArmor allows us to take away non essential ambient authority from all our processes, including the part of the file-system that should be considered as  global mutable parts of the filesystems from a user process perspective. Now in the place of where the public mutable state used to be, we drop in our own ‘private’ replacement, MinorFs. I won’t rehash MinorFS in this article as its extensively covered by my linux journal article, but basicaly it replaces the ‘public’ $TMP and $HOME  with a ‘private’ $TMP and $HOME, and allows for ways to pass by reference in order to do object oriented style pass by reference.

So now that we have our foundation (AppArmor, Fuse) in place, and have put our walls up (MinorFs), its time to look at our roof (E) again. Looking at our initial scenario of our user that wanted to share a secret document, the solution we buikd allows us to only have to trust our editor and our encryption tool with our secret. So instead of 20 programs of 50000 lines of code each we need to trust, there are only two. When these two programs would be implemented in E,  we would have to trust only 1000 lines of code per program instead of 50000 lines.  As a whole this would thus mean that our hypothetical trusted code base went down from one million lines of code to a mere two thousand lines of code., a factor of 500.

Two thousand lines of code other than one million are quite possible and affordable to audit for trust-ability and integrity. This means that our multi story least authority stack can provide us with a great and affordable way of building high integrity systems.  MinorFs is just a proof of concept, but it acts as an essential piece of glue for building high integrity systems. I hope people who read my article and/or this blog, and those people that were at one of my talks will think of MinorFs and of the multi story AppArmor+MinorFs+E approach that I advocated here and will apply the lessons learned in their own high integrity system designs.

Hypothesis: Reference stealing should be default (POLA) behaviour.

Those of you who have done programming in C++ are likely familiar with the auto_ptr smart pointer, or are familiar with coding standards that caution against the use of this type of smart pointer. With more and more parts of the new C++ standard making their way into C++ compilers, a new smart pointer type uniq_ptr is now available on the latest and hippest versions of some of the popular C++ compilers as part of the standard template library.

This new smart pointer type is said to deprecate the old auto_ptr smart pointer. If we combine the numerous C++ coding standards warning us against the use of auto_ptr with the C++ standard chromite deprecating the auto_ptr, it is hard not to jump to the conclusion that there must be something very wrong with this type of smart pointer.

In C++ we have a multiple ways to allocate objects and other data types. We have two ‘no-worries’ ways of allocating and one that requires the programmer to think about what he is doing as the language provides no default resource management for that type of allocation. An object or other data type can be allocated:

1. On the stack
2. Part of a composite object
3. On the heap.

When an object is allocated on the heap using new, the new invocation returns a raw pointer to the newly allocated object. To any seasoned C++ programmers a bare raw pointer in a random place in the code is an abomination, something that needs fixing.  You know the when you go to the beach and see morbidly obese man and woman in a way to tiny bathing-suites, and you think: “damn, please put some cloths on that” ?

Well thats how any C++ programmer (that is any good at his/her job) looks at raw heap pointers.  In C++ there are many cloths to choose from for raw heap pointers. The most elementary cloths are simple RAII objects that wrap a heap object pointer inside a stack object and thus controls the heap object its lifetime.  On the other side of the spectrum there is the shared_ptr, that as its name betrays is a shared reference counted smart pointer, the closest you will get to garbage collection in C++. Somewhere between those two we find a strange little creature called auto_ptr.

The auto_ptr is probably the simplest of all smart-pointers. It basically works like a RAII object with pointer semantics most of the time, but it has one funny little property that makes it behave rather odd.   Copy and assignment will under the hood result in a move instead of a copy. There are situations where its clear that this is desired behavior. For example when implementing the factory method or the abstract factory design patterns. These patterns can return an auto_ptr instead of a  raw pointer and the move will be exactly what anyone would want and would expect.

The problem however arises when an auto_ptr is used as an argument to a method. A method might steal our pointer by for example assigning it to a temporary or to a member variable of the invoked object.  I believe everyone can see there is a problem in there somewhere, but maybe, just maybe there is actually a glimpse of a solution to a problem hidden in there. A problem by the way that most programmers, especially C++ programmers don’t realize exists.

So lets have a look at the problem. When trying to build high integrity systems, there is one important principle that we must apply to every aspect of the system design and development. This principle is called the Principle Of Least Authority or POLA for short.  According to this principle we should, to state it simple, minimize the dependencies of each piece of code to the minimum amount possible. So what type of dependencies do we see in our pointer stealing example?  We see an object Bob with method foo and we see that this foo method takes some kind of reference to a Carol object. We see some object Alice with a reference to Bob and a reference to Carol that invokes :

• bob->foo(somewrapper(carol))

So what does bob->foo do with carol and more importantly, if we treat bob and its foo method like a black box, what would we want bob->foo to be able to do with carol? Basic usable security doctrine states that the default action should be the most secure one.  So what would be tha sane default most secure option here?  Basically this depends on the concurrency properties of Bob with respect to Alice. First lets assume that bob is a regular (non active) object. Than the least authority  option would be for somewrapper to be a RAII style temporary object that when going out of scope revokes access to carol. That way bobs access to carol would be scoped to the foo invocation.

Basically when building software using multi threading or multi process architecture, there are two mode’s of concurency we could use.

1. Shared state concurrency using raw threads, mutexes, semaphores, etc
2. Message passing concurrency using communicating event loops,  lock-free fifo’s, active objects etc.

As with many shared state solutions, when looking for high system integrity guards, the shared state solutions tend to be the solutions we should steer away from, so basically message passing concurrency should be the way to go here.

Now lets assume that we are using a message passing style of concurrency where alice, bob (and not necessarily carol) are active objects. What would be the sane least authority action for this invocation. The foo invocation scoping would make absolutely no sense in this scenario. The foo method will be handled asynchronously by bob. This means we do need to give bob a reference to Carol that will not be automatically revoked.  Given that carol is not necessarily an active object,  this reference would thus make carol an object that is shared between two concurrently running active objects.  Now given that sharing between two threads of operations implies quite some more authority and thus less implicit integrity for the overall system, the safest default operation may be that provided by what we considered to be wrong with auto_ptr in C++.  With single core computers becoming hard to find and the normal  amount of cores in a computer still going up, I believe its safe to say that we should assume modern programs will use some form of concurrent processing that according to POLA should probably be message passing concurrency. That is, the foo method should by default steal the reference to carol and allice should be explicit if she wants to retain her own copy of the reference to carol. This line of reasoning makes me come to the following hypothesis with respect to least authority.

• Reference stealing should be default (POLA) behavior for method invocation.

If this hypothesis is right, than the C++ auto_ptr flaw that the new C++ standard claims to fix by deprecating auto_ptr, might actually not be a flaw that needs fixing,  but a starting-point for,  in line with usable security doctrine,  making high integrity concerns into default behavior.

Dividing by uncertainty

I have a great deal of respect for the work done by ISECOM in their OSSTMM. Its overall a great and accessibly written document on doing security audits in a thorough and methodically. Its a document however, that suffers from the same, lets call it deterministic optimism that is paramount in the information security industry.  That is, an optimism that is a result of a failure to come to grasp with the nature of uncertainty.  While in this post I talk about the OSSTMM and how its failure to deal with uncertainty makes it overly optimistic about the True Protection that it helps to calculate, the OSSTMM is actually probably the best thing there is in this infosec sub field, so if even the OSSTMM doesn’t get this right, the whole subfield may be in for a black-swan event that will proof the point I am trying to make here.

So what is this uncertainty I talk about?  People tend to prefer hard numbers to fuzzy concepts like stochastics , but in many cases, certainly in information security, hard numbers are mostly impossible to get at, even when using a methodological approach like OSSTMM provides.  This means we can do one of two things:

1. Ignore the uncertainty of our numbers.
2. Work the uncertainty into our model.

A problem is that without an understanding of uncertainty, it is hard to know when its safe to opt for the first option and when it is not. If a variable, for example OSSTMM its Opsec(sum) variable has a level of uncertainty, a better representation than a single number could be a probability density function. A simplified variant of the probability density function is a simple histogram like the one below.

So instead of the hard number (10) we have a histogram of numbers and their probability.  So when does working with such histograms yield  significant different results than working with the hard numbers? Addition yields the same results, multiplication yields results that are generally close enough, but there is one operation where the histogram will give you a result that can, depending on your level of uncertainty and the proximity of your possible values to the dreaded zero value, and that is division.

In OSSTMM for example, the True Protection level is calculated by subtracting a security limit SecLimit from a base value. This means that if we  underestimate SecLimit we will end up being to optimistic about the true protection level.  And how is SecLimit calculated? Exactly, by dividing by an uncertain value. Worse, by dividing by an uncertain value that was itself calculated by dividing by an uncertain value.

To understand why this dividing by uncertainty can yield such different results when forgetting to take the uncertainty into account, we can device an artificial histogram to show how this happens. Lets say we have an uncertain number X with an expected value of 3. Now lets say the probability density histogram looks as follows:

• 10% probability of being 1
• 20% probability of being 2
• 40% probability of being 3
• 20% probability of being 4
• 10% probability of being 5

Now lets take the formula Y = 9 / (X^2). When working with the expected value of 3, the result would be 9/(3*3) = 9/9=1. Lets look what happens if we apply this same formula to our histogram:

• 10% probability of being 9
• 20% probability of being 2.25
• 40% probability of being 1
• 20% probability of being 0.5625
• 10% probability of being 0.36

Looking at the expected value of the result from the histogram, we see that this value ends up being almost twice the value we got when not taking the uncertainty into account. The lower numbers with low probabilities become the dominant factors in the result.

Note that I am in no way suggesting that these numbers will be anywhere as bad for the OSSTMM SecLimit variable, this depends greatly on the level of uncertainty of our input variables and its proximity to zero, but the above example does illustrate that not taking uncertainty into account when doing divisions can have big consequences.  In the case of OSSTMM, these consequences could make the calculated true protection level overly optimistic, what could in some cases lead to not implementing the level of security controls that this uncertainty would warrant.  This example shows us a very important and simple lesson about uncertainty. When dividing by an uncertain number, unless the uncertainty is small, be sure to include the uncertainty into your model or be prepared to get a result that is dangerously incorrect.

Programing language wishlist

My previous blog post was about garbage collection being anti-productive. Its my strong opinion that with respect to resource management, Java has got it completely wrong, C++ got it completely right, and languages with deterministic memory management (reference counting) like cPython come in as a good second to C++. I have to litle knowledge of C# to truly judge the productivity and scalability properties of its hybrid resource management model that at first glance seems to be a quite a bit better than Java Basically. I would say (taking into respect what I heard and read about C#) with respect to resource management:

• +2 for C++
• +1 for python
• +/- for C#
• -2 for Java

But I will be the first to admit that Java,  and especially its more secure little brother Joe-E, has many merits on other fronts, while C++ definitely has its weaknesses. Looking at the wide range of programing languages available, and looking at their features and miss-features,  there doesn’t seem to be a single language that doesn’t have at least one ingredient that is incompatible with my personal taste. This made me think that if I could order a programing language the way I can order a pizza: Tuna pizza,  double garlic, pickled green peppers , hold the onions.  What toping would I choose for my programming language? Its likely that my taste in programing language is just as disgusting to you as my taste  in pizza is to many of my friends My priorities basically are scalable high productivity and high integrity, if you have other priorities, your pizza will likely get quite a different combination of toppings.

Perl’s expressiveness and built-in regular expression support.

While I try to not use Perl anymore, I started my career as a Unix system administrator doing a lot of Perl, and have done many smaller and mid sized projects in Perl. While I’m no longer the big Perl fan I used to be, Perl expressiveness  keeps pulling me back to Perl if I need to build something small in a hurry. No language beats Perl with respect to how little code or how little time it takes to build something when in a hurry. Combined with the built-in regex support that is really integrated well in the language, this is a toping I would love to have onh my programing language pizza.

C++ its full encapsulation of resources, add some automated  resource management.

As I showed in my previous blog post on resource management, C++ RAII allows resources to be fully encapsulated without braking encapsulation or burdening polymorphism. This property in C++ comes at the price of not having automated memory management. I’d like to have my cake and eat it to here. C++ its full encapsulation of resources, but with some more syntactic sugar for RAII to further boost productivity.

C++ its generics and TMP support

Nothing boosts productivity like being smart about the  ’Dont Repeat Yourself’ or DRY principle. Nothing spells DRY like C++ generics and template meta programing in C++.

C++/Python’s operator overloading

While operator overloading to many is just syntactic sugar, it to me is more like syntactic honey, making things do down much smoother in many cases.  Function objects (objects overloading the ‘()’ operator) are a great alternative to old-fashioned C callback functions.   In C++ cast operators can safe you much work when refactoring. Operator overloading on their own will boost your productivity, but when combined with generics this boost is multiplied.

Pythons Strong, Safe but property based type system.

Strong and safe type systems are great, but sometimes they get in your way. In python a strong and safe type system is combined with property based compatibility.

Message passing concurrency only, safe the raw threads.

Where shared mutable state can get you into trouble, you haven’t seen any real trouble until you have tried doing large scale development with raw threads. Raw threads give you shared state concurrency with all the troubles that sharing state brings with it. These problems can really kill your productivity and create hard to track down bugs. Message passing concurrency gives a less problematic alternative.  I would like my ideal language not to supply me with raw threads, but hide these (and lock free containers) these behind an abstraction layer that  provides message passing concurrency instead.

No class variable support.

As with shared state concurrency, class variables are shared essentially shared mutable state and shared mutable state is something that can be tricky to work with. There are a few basic rules for working with shared mutable state that come from object capability theory and from basics about testability of code. In essence class variables are not compatible with these rules.

No pointer arithmetic support

This for me is the thing I dislike most about C++. Pointer arithmetic will get you into trouble. Its a tempting thing to use in C++, and thats why I don’t want it in my ideal language.  Its like one of those fattening ingredients on your pizza that you know is bad for you, but you have a graving for it so you take it.

No reflection or typeof

Reflection and typeof will let you get away with bad design, it will let you be more productive for the short haul, but bad design will in the end make you regret you took a shortcut and will make you pay for taking the shortcut many times over.

Fully ocap with persistent object support

The E-language, a language that I have used for smaller personal projects (Its quite impossible to convince my co-workers about this great but exotic language is worth investing serious time in), combines being fully OCAP (a sane subset of OO for building high integrity systems) and has support for persistent applications that survive system reboots without loosing state. The E-Language is fully compatible with my MinorFs project and allows for fitting least authority design into a multi granularity system design from the OS level up.  My Ideal language would take quite a few things from E. Its an amazing litle language that has gotten way to little press.

Co-worker compatible

A language could have all the ingredients above, a real project requires multiple people working on it. This means that the ideal language, other than E should be compatible with the often conservative views of co-workers with respect to investing in new technology.  The ideal language should be similar enough to languages my co-workers know (Java,C++,Python,C#,Delphi,Perl) to be acceptable to the conservative mind.

The above list to a point is just personal taste, but I believe its most about personal taste in priorities that for me are productivity and high integrity. My ideal language would allow me and my co-workers to write large scale high integrity systems with the highest possible productivity.

Why garbage collection is anti-productive.

While the non GC language C(++) is still the most popular language in the world (at least if we count C and C++ as the same language, if we look at them as separate languages, Java is the most popular language), GC languages as a whole currently make up a larger share than non GC languages, and new languages seem to build on virtual machine environments that mandate GC. It thus would be good to look at what GC has gotten us so far.

Lets start by looking what GC is for. GC is basically a form of automated resource management for one specific type of resource: Random Access Memory or RAM. In computer programming there are multiple types of resources: RAM, open file handles, locks, threads, etc,etc. GC automates resource management for just one of these resource types: memory.

This means that in a GC language, there are two types of resource, memory (a resource you should no longer worry about), and anything else, for what you should do some manual resource management.

Looking at this, it would seem that GC is a pretty good deal. You don’t have to worry anymore about resource management for memory so you gain productivity there, and for other resources nothing changes. But it is this last assumption that ends up as a giant mistake.

To understand why thinking nothing changes for non-memory resource management in GC languages, lets have a look of how a C++ programmer would do resource management on raw memory.

class RawMemoryWrapper {
char *mRaw;
public:
RawMemoryWrapper(size_t bytes):
mRaw(new char[bytes]){}
~RawMemoryWrapper(){ delete []mRaw;}
operator char * () { return mRaw;}
};

We see a special class (refered in C++ as a RAII class) that wraps the resource. If a RAII class is created, the constructor acquires the resource. Hence the name ‘Resource Acquisition Is Initialization’ or RAII. If the RAII object goes out of scope, its destructor is called and the resource is released auto-magically. A third construct that is not actualy part of the RAII idiom but is used often together with RAII is a C++ cast operator that allows RAII classes to be used as drop in replacement for the resource they wrap. RAII helps wrapping any resource we might want to use in a fully encapsulating way.  You can make composite classes with RAII member variables or without them, there is nothing transitive about resources in RAII, what is a good thing from an OO perspective.

Although a RAII class isn’t that big, for memory its clear there IS overhead in having to write RAII classes when compared to GC.  From this we might conclude that this alone makes GC a winner, after-all, if at one front we win and at an other front we don’t lose we would have won. This assumption however is wrong for one elementary property of GC. GC works with the abstraction of eternal objects.  If an object goes out of scope, basically nothing happens. No destructor is called, no memory is freed. In most GC implementations, even when the memory the object held is freed, no destructor is called. This property of GC basically means that GC exludes the posibility for using the RAII idiom on non memory resources.

So what is the cost of not being able to use RAII for non memory resources in GC languages?  This cost can be expressed with a thing that shocked me when I grasped its impications:

• Without RAII, ‘being a resource’  becomes transitive to composition.

Once you grasp what this means for object orientation, you will be as shocked as me by the popularity of GC and about the fact that the productivity gain that GC offers is so clearly a myth. To me it has become absolutely clear that GC does not only not improve resource management related productivity, it actually decreases resource management related productivity in such a dramatic way that the persistence of the productivity myth is absolutely shocking.

Imagine that you have a relatively large program. You will be pretty sure to have many resources in need of manual management.  These resources with RAII classes could be managed once down deep in some library code. In a large program compositional patterns may easily go 10 or 20 layers deep, and at each of these layers, without RAII, given the transitive nature of being a resource,  the result of this would be that there should hardly be any non resource classes within the higher abstraction layers of such a project.

So we started off trying to reduce the amount of manual resource management, but what happened? We exploded the amount of manual resource management and in the process broke the encapsulation principle good of OO programing  practice. A user of a class should have no knowledge if the class being used happens to be composed using non memory resources.  We can clearly state that:

• GC breaks OO encapsulation.

But it doesn’t end there.It gets worse.  Given that we are using OO design and polymorphism,  an interface of a class should not change depending on class implementation.  So if we can conceive of the idea that a class could either be implemented without a composition that used non-memory resources, but also one that does, than given the  nature of being a resource in GC languages, we MUST when designing an interface assume there might be an implementation that is composed using non-memory resources.This means that we need to treat any polymorphic interface as a resource even if its implementation isn’t composed using any actual non memory resources. We can thus come to the following conclusion:

• GC breaks OO polymorphism.

This in term means that our already larger need for manual resource management expands once more to even larger dimensions.

So lets look at all the facts once more:

• Without RAII, ‘being a resource’  becomes transitive to composition.
• GC breaks OO encapsulation.
• GC breaks OO polymorphism.
• As a result GC ‘explodes’ the need for manual resource management and thus reduces productivity.

With GC having started out as a way to reduce manual resource management and increase productivity, its pretty clear that GC has ended up being one big mistake.

Capibara.com and op.nu

Recently I received a message that my capibara.com domain required renewal. Capibara.com is the only remaining domain of the first 3 domains I owned and it has quite a history, but in recent years I basicaly forgot about this domain.

The renewal message made me remember the history of this domain, and made me think that I shouldn’t allow this domain that did play a modest but important role in the history of the world wide web to waste away.  So I decided to put wordpress up on this domain and this is my first blog. It seems only apropriate to dedicate my first blog post to the history of this domain.

The history of capibara.com (and the now long expired op.nu domain) started 14 years ago. At that time domain hosting wasn’t as cheap as it is today, in fact it was shamefully expensive.  With capibara.com I tried to change that. Fortunately in those days HTTP hosting had no virtual hosting, so my capibara.com server at that time was running on its own IP.  I created a simple index.cgi in Perl that looked at the requested domain, looked up an URL in an on-disk hashmap belonging with this domain, and generated a single frame page that hid the fact that the page wasn’t hosted at the requested domain. I created a script where people could register their domain and long url with capibara.com, and capibara.com would map the domain to the url. Capibara.com didn’t solve everything, but an other free service, granitecanyon that offered a free DNS service, and multiple providers of free web space (with long ugly url’s) combined to make it possible to get your own domain for just the domain registry costs.

For people for who this was still to much, I had the domain op.nu registered, and at granitecanyon had a wildcart domain entry for this domain. People could claim a free sub domain of op.nu.

For years this service helped people with litle money to be able to affort their own domain. At the end there were many thousands of domains mapper using the capibara.com url mapping service. But than things went badly. The hosting company that hosted capibara.com and tolerated the domain hosting service went bust, and as you may understand, other companies were unwilling to accept capibara.com on their servers.

At that same time ADSL was starting to pick up, but most ADSL providers didn’t deliver good quality of service on these lines, and Linux was getting prety popular already. The CDUCK project was born to fill the gap that the demise of the capibara free url cloacking service left. CDUCK was a distributed version of the services provided by granitecanyon and capibara, one that people could run at home.

With hosting programs now affordable to most and CDUCK orphaned as project on sourceforge, there is not much left of my original capibara.com initiative, neither is it needed anymore, but I do feel that capibara.com has played a modest yet important role in the history of the world wide web, one that makes this domain deserving of a second life as my personal blog space.