New CSS skin – NUX

In my latest release of Phosphorus Five, there’s now a new default Micro CSS skin called “NUX”. Its name implies “!UX”, since I have tried to make it very much minimalistic. The idea is to create a skin, completely free from colours and other types of “bling”, to allow for higher readability and more easily navigate your app’s actual content. You can see a screenshot of it in the above right corner. If you’d like to see the screenshot fullscreen, feel free to simply click the image.

Below is an example of how Hyper IDE looks like when using this new skin.

I feel the content of your apps becomes more visible, and emphasised, simply since there are less colours in this skin. It also significantly increases the sizes of clickable elements, such as radio buttons and checkboxes. Which I feel makes it more usable for smaller devices, such as tablets and phones.

In addition to the NUX skin, I have also created a couple of new convenience widgets, for more easily creating checkboxes and groups of radio buttons. And I have also significantly improved the rendering of Camphora Apps on smaller devices, and in general. Below is a screenshot of the “sales” app.

Feel free to check it out 🙂

Encapsulating your system through the file system

To a seasoned developer, the above sentence might sound like a Haiku. Something a computer might have generated automatically by following the rules of grammar, to create random sentences, incapable of communicating meaning. However, here’s an example of what I mean. Feel free to browse the “imp” folder to understand what I mean.

Hyperlambda has no namespaces, no classes, no private or protected keywords – Still, I’d argue it allows you to create much better encapsulation of your code. Simply since I can evaluate files as if they were functions. I can pass in arguments to my files, and do something resembling the following.

eval ("/backend/tickets/model/get-ticket.hl", id:5)

The above code is of course pseudo code, however its actual implementation is surprisingly similar in syntax. You can see it below. What it returns, is pretty obvious I assume.

micro.evaluate.file:@INGEN/imp/backend/tickets/model/get-ticket.hl
  id:x:/../*/id?value

Let’s use another example, how I build the toolbar in the top right corner of the system.

  micro.widgets.file
    class:strip themepark-top-buttons
    folder:@INGEN/imp/common/toolbar-buttons/

It doesn’t require a rocket scientist to understand what that code does, or? Here is the folder it’s pointing to. If you take a look at one of its buttons, you’ll see something like this.

button
  innerValue:@"<span class=""icon-key""></span>"
  oninit
    whoami
    if:x:/@whoami/*/default?value
      !=:bool:true
      delete-widget:x:/../*/_event?value
  onclick
    desktop.authentication.login

Even without its comments, I believe most 10 year olds would instantly understand what the above code does. Below is a screenshot of what happens when I click the button. Notice, the screenshot below, including the button to trigger it, was created with no more than 9 lines of code.

Creating a new toolbar button, is as easy as creating a new Hyperlambda file, and putting it in the “/imp/common/toolbar-buttons/” folder. How many lines of code would you need to create something like the above in C#? Let’s try to imagine that as a thought experiment …

First we’d need support for dynamically loading up controls to populate our toolbar. This would probably require at least some 3-4 different design patterns, such as an Abstract Factory, Factory Methods, etc. Each button would have to implement some interface. Then we’d have to create some sort of JavaScript frontend part, allowing us to actually click the button, and creating an Ajax HTTP request, which does whatever the button does once clicked. A qualified guess would be.

  1. Factory methods and classes to dynamically populate toolbar ==> easily 300 – 500 lines of code
  2. Interface for buttons ==> probably 50 – 100 lines of code
  3. Each button’s C# file ==> between 50 and 500 lines of code, depending upon what the button does
  4. JavaScript to create our Ajax HTTP requests ==> At least 100-200 lines of code for each button

And we’d end up with a solution where even the smallest change in one of our parts, often would result in cascading effects, forcing us to change every single button, and every single part of our solution. For instance, adding another argument to our interface method results in having to change every single code file. Even with a conservative estimate, the above would result in 800 lines of code for three simple buttons – Possibly as much as 2,900 lines of code. I don’t think many seasoned developers would object to these numbers. If you don’t count the comments, the Hyperlambda equivalent becomes

  1. 3 lines of code for the toolbar
  2. 35 lines of code for all buttons

That’s 38 lines of code! Versus 800-2,900 lines of code. If we were to use our worst possible estimate from our C# solution, it would require 82 times as much code! Since the amount of cognitive energy you have to apply to a software project is directly proportional to the number of lines of code your solution contains, this results in that a pure C# solution would require possibly 82 times as much cognitive energy to be understood as its Hyperlambda equivalent. This translates into that if you require 5 minutes to understand the Hyperlambda code, you’d need almost 7 hours to understand the C# solution.

  1. Hyperlambda ==> 5 minutes, 38 lines of code
  2. C# ==> ~7 hours, 2,900 lines of code

All of these numbers I played around with here are numbers that most seasoned system developers could easily agree with me in regards to, so I cannot be blamed for having skewed the numbers in any ways. In addition to that purely mathematically the C# solution would require 82 times as much energy, it also contains 2 different programming languages (JavaScript and C#). On top of that, it contains several design patterns, to facilitate for creating a plugin architecture capable of dynamically loading buttons. Possibly some sort of configuration file (JSON/XML?) too, to declare which buttons you want to load. Hence, the C# solution would literally be scattered with dependencies, making even the tiniest change in any one of those files, possibly cascade into all 2,900 lines of code – Forcing you to effectively rewrite your entire solution, due to adding a single additional parameter to the instantiation of your buttons (style argument …?)

While for the Hyperlambda solution, all you’d have to do would be to change the [micro.widgets.file] invocation from above, to resemble the following.

micro.widgets.file
  class:strip themepark-top-buttons
  folder:@INGEN/imp/common/toolbar-buttons/
    style:"padding:3rem;"

One additional line of code, compared to spending an entire afternoon rewriting interfaces and implementations. The above addition to my toolbar changed my buttons from this …

… to this …

It doesn’t take a rocket scientist to understand the advantage here … 😉

So while the C# developer is changing his interface to pass in style arguments to the instantiation of his buttons, editing literally every single code file in his solution, spending the better parts of an entire afternoon – Assuming he understands the code in the first place, and don’t need 5 days to simply understand the class hierarchy. I’ve moved on to the “next problem” literally after 2 seconds!!

Didn’t like my style argument? Well, OK – Let’s create an [onmouseover] event, that displays the title attribute in an “information bubble” window for our buttons.

micro.widgets.file
  class:strip themepark-top-buttons
  folder:@INGEN/imp/common/toolbar-buttons/
    onmouseover
      get-widget-property:x:/../*/_event?value
        title
      micro.windows.info:x:/-/*/*?value

4 additional lines of code, and all of a sudden every single button in our toolbar displays something resembling the following when the mouse is hovered over the button.

How many hundreds or thousands of additional lines of C#/JavaScript/“whatever” do you think the above would need in literally any other traditional programming language …?

In case you didn’t like my above example, let’s do something more “hard core”. In fact, let’s invoke a web service, and let’s PGP encrypt our invocation and cryptographically sign our invocation. And just for the fun of it, let’s have the client supply the code that we evaluate on the web service endpoint. For then to have the web service verify the signature of our PGP key used to cryptographically sign our invocation, and allow specific PGP key signers to have “extended rights” during evaluation on our server.

/*
 * Invoking web service, passing in Hyperlambda to
 * be evaluated on my server.
 */
micro.web-service.invoke:"https://home.gaiasoul.com/hypereval/lambda-ws"
  fingerprint:ea630b9a61448b7f647045c210e1e050ce6feb1d
  encrypt:true
  lambda
    .foo:This code was evaluated on the web service endpoint
    split:x:/-?value
      =:" "
    return:x:/-/*?name

8 lines of code. In fact, I am so certain about that you’d need thousands of lines of code to be able to do something even closely resembling the above in C#, that I am willing to bet a kidney on that you can’t without at least some 5-10,000 lines of C# code! 8 lines of code replacing 10,000 lines of code. That’s an increase of 1,250 times the cognitive load to implement the above in C#. And for the record, the above code is working code. If you download Phosphorus Five, you can immediately paste it into “Hypereval” and execute it. It will evaluate the code inside of its [lambda] node on my server for the record!

If it takes 5 minutes to implement the above in Hyperlambda, it would require 104 hours to do it in C#!

A seasoned software architect and software developer might laugh of Hyperlambda, proclaiming it’s not a “real” programming language, since it doesn’t contain x, y or z. However, none of that matters, because when somebody skilled enough to understand it puts themselves behind the controls of that thing, they’re doing warp 10, and have crossed half the galaxy and returned – Before the competition has even managed to start up his steam engines …

It has no wheels, no visible engines, and no gearbox – How can you claim it’s a vehicle …?

Well, since a picture is worth more than a thousand words, I’ll finish this article with a picture and a simple question …

How would you want to travel today Sir …?

InGen Dinosaur Theme Park system, adding speech synthesis and guided tours

I put in a handful of additional coding hours today in the InGen Dinosaur Theme Park Software System,  and below is the results so far. So far I have spent about 10-12 hours of actively coding on the system, yet it’s still very feature rich, with among other things guided tours localised to “whatever” language using your browser’s speech synthesis features, and Google Translate to translate the tour into whatever language you want to have your tour in.

It features integrated QR codes that are automatically created as URLs leading to such tours, allowing the theme park to plaster these around at relevant locations, to provide their guests with guided tours, in as many languages as they want to. Watch the video below demonstrating the system.

I’ll probably add some additional 5-10 hours of coding before I consider it done. However, if you want to try it out, you can download its zip file, and simply upload it into any existing Phosphorus Five installation.

The InGen Dinosaur Theme Park System

So, the “Shitty Programming” contest was being held last night, and both of the participants were given the same specification, which basically was to implement a new IT system for “Jurassic Park”. Obviously, none of us could finish, since the system required sensors for checking if doors were open, and cameras in cages, etc, etc, etc – But below you can see my results roughly the way it was when I was finished implementing it.

Notice – I want to emphasise, that I have spent a couple of additional hours cleaning up the system’s code and UX after the contest was over. However, fundamentally I had most of these features already implemented after 3 hours of coding. And may I add, that was an extremely stressing coding act, since we were both constantly being watched by 3-400 people live, in a stream, and asked questions every now and then about our choices, etc. You can download the InGen Theme Park system here. In fact, if you have a Phosphorus Five installation somewhere, you can install it through the Desktop module, by clicking the “Upload module” button – At which point the system will be installed immediately, create its own database, and be instantly ready to start using.

Shitty programming death match TONIGHT!!

Roughly 5 hours from now, 8PM EST, the 22nd of June 2018, the “Shitty Programming contest” will go down. It’s me against FascinatedBox, and it’s a programming contest to DEATH! 😉

Read more about the contest in general here. If you’d like to watch it, you can check out what time it is in your local timezone here. As of from this writing, it’s happening roughly 5 hours from now! In Cyprus it will occur 03:00 (at night) the 23rd of June. In Norway it’ll occur 02:00 the 23rd of June.

Here is where the show goes down!! (Warning, there is a “mature language” disclaimer on the channel, which I guess explains what type of contest this will be)

Get ready to unleash the shittiest double shitty programming contest this world has **ever seen**!! 😀

What can I say …?

UNLEASH THE SHITSTORM!! 😀

Psst, here’s a fundraiser for a good cause if you have some dough to spare … 🙂

Invoking a PGP encrypted lambda web service with 5 lines of code

I have just now released a new version of Phosphorus Five, where one of the major new features is that it is now dead simple to invoke and create “PGP enabled lambda web services”. A “lambda web service” is basically a web service where the client supplies the code that is to be executed in the web service endpoint. Since this is done through a cryptographically signed HTTP request, it’s actually highly secure, since this allows you to whitelist only a bunch of specific pre-defined PGP keys, and decide which Active Events the caller is legally allowed to execute on your server. You can also control which Active Events a caller is allowed to execute according to which PGP key was used to cryptographically sign the invocation.

If you’d like to test this, you can immediately do so in fact, by first downloading the source code for Phosphorus Five, get it up running, start “Hypereval”, and start invoking lambda web services on my private server. My server is at home.gaiasoul.com, and the first thing you’ll need to do, is to import my public PGP key into your own server. This can be done by pasting the following code into “Hypereval”, and click the “lightning” button, which will execute your code.

p5.http.get:"https://home.gaiasoul.com/micro/pgp"
p5.crypto.pgp-keys.public.import:x:/-/**/content?value
eval-x:x:/+/*/*/*/*
create-widgets
  micro.widgets.modal
    widgets
      pre
        innerValue:x:/../*/p5.crypto.pgp-keys.public.import/0?name

The above code will result in something resembling the following.

My server’s PGP key is as follows.

ea630b9a61448b7f647045c210e1e050ce6feb1d

The above snippet will import my public PGP key into your server, and display the “fingerprint” for my server’s PGP key in a modal window. This is important to later use as you invoke web services on my server. After you have copied the above fingerprint, just click anywhere outside of the modal window to close it, and replace your code in Hypereval with the following code.

/*
 * Invoking web service, passing in Hyperlambda to
 * be evaluated on my server.
 */
micro.web-service.invoke:"https://home.gaiasoul.com/hypereval/lambda-ws"
  fingerprint:ea630b9a61448b7f647045c210e1e050ce6feb1d
  lambda
    /*
     * This code will be executed on my web server!
     */
    .foo:This code was evaluated on the web service endpoint
    split:x:/-?value
      =:" "
    return:x:/-/*?name
/*
 * Displaying a modal widget with the result of
 * the web service invocation.
 */
lambda2hyper:x:/../*/micro.web-service.invoke/**/content/*
create-widgets
  micro.widgets.modal
    widgets
      pre
        innerValue:x:/@lambda2hyper?value

I want to emphasise that the above Hyperlambda was actually executed on *MY WEB SERVER*! And this was done *SECURELY*! The way I have setup my web service endpoint, is to simply use the default implementation. This is a “Hypereval” page snippet, which you can visit with a normal browser by going to home.gaiasoul.com/hypereval/lambda-ws. This will create an HTTP GET request, which will load the web service endpoint’s GUI, allowing you to play around with code, and execute it on my server, using a CodeMirror code editor. Click CTRL+SPACE or CMD+SPACE (OSX) in it to trigger AutoCompletion.

If you create an HTTP POST request towards the same URL, this will assume you want to create a web service invocation towards my server, and act accordingly. The above [micro.web-service.invoke] Active Event is simply a convenience wrapper to create a client connection towards such a web service endpoint.

Implementation

What is actually occurring beneath the hoods, is that a MIME envelope is created internally. Then the entire MIME message is cryptographically signed, using your server’s private PGP key. This signature is verified in my server when the MIME message is parsed, and only if the signature verifies, the execution of your code is allowed. Before it is being executed though, if will invoke [hypereval.lambda-ws.authorize-fingerprint], to check if the PGP key that signed the invocation has “extended rights”. This allows me to “whitelist” things that I normally don’t consider safe only for a handful of PGP keys which I trust comes from sources that I know I can trust. For instance, if a friend of mine had a Phosphorus Five server, and I trust his server, I could allow him to create SQL select invocations, etc – While a normal request, originating from an untrusted source, does not have these rights. I can also choose to entirely shut down access for all keys except a bunch of pre-defined “whitelisted” keys if I want to.

When the web service invocation returns to the client, it will verify that the returned request was cryptographically signed with the [fingerprint] I supplied during invocation, and if not, it will not accept the invocation, but rather throw an exception. This completely eliminate any “man in the middle” attacks, assuming I have verified the PGP fingerprint of the server I am invoking me request towards.

If I wanted to encrypt the entire invocation, I could add a simple [encrypt] node, and set its value to boolean “true”, at which point both the request from my client, and the response from the server, will be encrypted using PGP cryptography. Below is a code snippet demonstrating this.

/*
 * Invoking web service, passing in Hyperlambda to
 * be evaluated on my server.
 */
micro.web-service.invoke:"https://home.gaiasoul.com/hypereval/lambda-ws"
  fingerprint:ea630b9a61448b7f647045c210e1e050ce6feb1d
  encrypt:true
  lambda
    /*
     * This code will be executed on my web server!
     */
    .foo:This code was evaluated on the web service endpoint
    split:x:/-?value
      =:" "
    return:x:/-/*?name
/*
 * Displaying a modal widget with the result of
 * the web service invocation.
 */
lambda2hyper:x:/../*/micro.web-service.invoke/**/content/*
create-widgets
  micro.widgets.modal
    widgets
      pre
        innerValue:x:/@lambda2hyper?value

Notice, the only difference between my first code and this last snippet is the [encrypt] argument, having a boolean “true” value. This will encrypt the request for my server’s PGP key, and when the response is created on my server, it will encrypt the response for your PGP key. All of this happening “for free”, without you having to do anything to implement it.

I can also submit multiple [lambda] nodes, which will be executed in order of appearance, and I can also supply an ID to each [lambda] node, which helps me identify each snippet evaluates on the server. Below is an example.

micro.web-service.invoke:"https://home.gaiasoul.com/hypereval/lambda-ws"
  fingerprint:ea630b9a61448b7f647045c210e1e050ce6feb1d
  lambda:foo-1
    .foo:This code was evaluated on the web service endpoint
    split:x:/-?value
      =:" "
    return:x:/-/*?name
  lambda:foo-2
    +:int:7
      _:14
    return:x:/-?value
lambda2hyper:x:/../*/micro.web-service.invoke/**/result
create-widgets
  micro.widgets.modal
    widgets
      pre
        innerValue:x:/@lambda2hyper?value

The above will display as the following when being executed.

In addition I can also pass files back and forth between my web service endpoint and my client. I can also reference these files from inside my [lambda] nodes, but this is another article later. However, I want to emphasise that files passed back and forth between the client and the server is serialised directly from disc to the HTTP connection, and hence never loaded up into memory, allowing me to create “ridiculously large” HTTP web service requests, without exhausting neither the server nor the client in any ways. Still everything transmitted is easily PGP encrypted, by simply adding a simple argument to my invocation.

Internally what occurs, is that I actually create a MIME request and response, which internally is usingMimeKit, which is the by far fastest MIME parser in existence out there for .Net/Mono. I have tested it myself and serialised 5,000 5KB small images, and it was able to encrypt and decrypt 5,000 images in roughly 10-12 seconds. And since the Open PGP standard dictates that PGP encrypted envelopes will actually be compressed (think “zipped”), this has the additional bonus of significantly reducing the bandwidth consumption both ways. In general, you can expect the request and response to be some 5-10 percent in size compared to transmitting the files as “raw” (unencrypted) data.

Creating your own web service endpoint

If you’d like to create your own web service endpoint, you’ll basically only have to open up for a couple of URLs using “Peeples”. This allows access to the “guest” account to access the URLs necessary to evaluate a web service invocation on your server. Notice, you can still restrict who actually have access to your server though, by changing the settings for your lambda web service endpoint. Below is what you’ll have to paste into the Access object list of your “Peeples” module.

*
  p5.module.allow:/modules/hypereval/lambda-ws/
    exact:true
*
  p5.module.allow:/modules/micro/pgp/
    exact:true

The first part is to enable your actual web service endpoint, and the second part is to enable clients to download your PGP key. This should look like the following in your Peeples module.

Make sure you click “Save” after having pasted the above into Peeples. To change the settings of your web service endpoint, open up Hypereval and load the “lambda-ws” page, and read the documentation for it.

Download Phosphorus Five here.

Playing around with MIME/PGP lambda web services

In the upcoming release of Phosphorus Five, one of the things I have given great attention to, is its MIME and PGP parts. One of its really nifty use cases is the ability to invoke cryptographically signed and encrypted web services passing in MIME envelopes as the body of your request. This has a lot of interesting use cases, such as making it possible to demand that the invoker of your web service has cryptographically signed his invocation using a “whitelisted” PGP key – Resulting in a web service that can only be invoked by an authorised client. Or to transmit multiple Hyperlambda snippets, that are evaluated in order of appearance. Or to transmit file attachments together with your web service invocation, having access to these file attachments from your own (client) Hyperlambda, etc, etc, etc. If you have no idea what a “lambda web service is”, you can read one of my DZone articles about it here.

In the upcoming 8.4 release I will have a Hypereval “page” snippet, that actually acts as a web service endpoint, encapsulating a lot of interesting features down this alley. Notice, the code examples here requires that you have at least version 8.4 of Phosphorus Five, which is not yet released as of this writing. Anyway, let us look at what we can do, as 8.4 is released.

A simple HTTP POST web service invocation

So let us first create a simple (non PGP) web service invocation. Notice, if you’ve downloaded version 8.4, you’ll need to use e.g. “Peeples” to grant module access to the “/hypereval/lambda-ws” page to the guest account to be able to use this code.

/*
 * Creating a normal lambda web service invocation.
 */
p5.http.post:"http://127.0.0.1:8080/hypereval/lambda-ws"
  Content-Type:application/x-hyperlambda
  content
    +:int:7
      *:int:20
        _:10
    return:x:/@+?value

The above Hyperlambda snippet will create an (unencrypted) HTTP POST request going towards your Hypereval “lambda-ws” page, and pass in a Hyperlambda snippet, that simply multiples 20 with 10, for then to add 7 to the result, and echos the results (207) back to the client. A slightly more complex version can be imagined using the following code.

p5.http.post:"http://127.0.0.1:8080/hypereval/lambda-ws"
  Content-Type:application/x-hyperlambda
  content
    split:Thomas Hansen is pretty cool
      =:" "
    add:x:/+
      src:x:/@split/*?name
    return

The above will result in something resembling the following.

p5.http.post
  result:"http://127.0.0.1:8080/hypereval/lambda-ws"
    status:OK
    Status-Description:OK
    Content-Type:application/x-hyperlambda
    Last-Modified:date:"2018-06-19T13:17:04.998"
    Server:Mono.WebServer.XSP/4.4.0.0 MacOSX
    Date:"Tue, 19 Jun 2018 10:17:04 GMT"
    Cache-Control:private
    Set-Cookie:ASP.NET_SessionId=34F7802887AEA977F431B3DA; path=/; HttpOnly
    Connection:close
    content
      Thomas
      Hansen
      is
      pretty
      cool

Basically passing in Hyperlambda code, that splits a sentence (on the server may I add), as a web service invocation, and returns each word back to caller, as the body of the HTTP response. The reasons why this is pretty safe and secure, is because as I evaluate the Hyperlambda supplied by the client, I am using [eval-whitelist], which allows me to only allow for a (secure) subset of my server’s Active Event vocabulary. Hence, no malicious code constructs are possible to create that somehow changes the state of my server. What vocabulary you want to whitelist is easily configurable.

Encrypting and signing your web service invocation using PGP

This is where it gets interesting. First let’s have a look at some code that does something similar to our above code, with two crucial differences. First of all it encrypts our code. Secondly it cryptographically signs your code using a private PGP key.

/*
 * Creating a cryptographically secured lambda web service invocation.
 */
p5.http.post:"http://127.0.0.1:8080/hypereval/lambda-ws"
  .onrequest
    .p5.mime.serialize-to-stream
      multipart:mixed
        encrypt
          fingerprint:83bfcbe0235f90e0a67bb865863a22266f9ef39a
        sign
        application:x-hyperlambda
          content
            split:Thomas Hansen is pretty cool
              =:" "
            add:x:/+
              src:x:/@split/*?name
            return
  .onresponse
    .p5.mime.load-from-stream

The above code will create a MIME envelope of our “string split” Hyperlambda. Then it will encrypt the code for then to cryptographically sign the message. The code will be cryptographically signed by your server’s PGP key, and it will be encrypted for the PGP key having the fingerprint of “83bfcbe0235f90e0a67bb865863a22266f9ef39a”. The latter of course must match the PGP key of the server you are transmitting your request to. Which you can read about how to acquire here.

This creates a couple of really interesting side effects. First of all, all of a sudden we have a guarantee of that the request originated by the client that had access to the private PGP key that was used to sign the envelope. Since the web service endpoint allows us to create a “white list” of accepted PGP keys, this allows us to only accept invocations from clients we know for a fact we can trust – Assuming your private PGP key hasn’t been compromised somehow of course.

In addition, it becomes literally impossible for an adversary to spy on your web service invocations. For code, this might not be that crucial. However, the MIME features of Phosphorus Five also allows you to transfer data and files, making this “tiny little detail” important. In addition, it also completely eliminates the problem with a “Man In The Middle” acting as a malicious proxy, tricking you into believing you have a secure SSL context, by giving you a fake certificate. In fact, even in the middle of a “Man In the Middle” attack, you can still have a cryptographic guarantee of that you have a securely encrypted communication channel – Assuming your PGP keys are correct though.

It also prevents “tampering” with the message, since if such a thing occurred, the PGP signature would no longer verify, and the invocation would never be evaluated. And since this all happens “automatically” for you, and the code is returned “semantically”, there is zero overhead for you as an application developer to implement this. In fact, here is what is returned from the web service after invocation.

p5.http.post
  result:"http://127.0.0.1:8080/hypereval/lambda-ws"
    status:OK
    Status-Description:OK
    Content-Type:text/html
    Last-Modified:date:"2018-06-19T13:24:19.390"
    Server:Mono.WebServer.XSP/4.4.0.0 MacOSX
    Date:"Tue, 19 Jun 2018 10:24:19 GMT"
    Cache-Control:private
    Set-Cookie:ASP.NET_SessionId=33FDE6A445F8D46F03602E74; path=/; HttpOnly
    Connection:close
    content
      multipart:signed
        signature
          foo@bar.com:bool:true
            fingerprint:83bfcbe0235f90e0a67bb865863a22266f9ef39a
        multipart:mixed
          application:x-hyperlambda
            content
              Thomas
              Hansen
              is
              pretty
              cool
        application:pgp-signature
          Content-Disposition:attachment; filename=signature.pgp
          Content-Transfer-Encoding:7bit
          content:blob: ... a long piece of PGP signature blob ...

At which point you can retrieve your Hyperlambda results with a simple x-pression. If the web service endpoint receives something that is cryptographically signed, it will encrypt the return value for that same PGP key, and cryptographically sign it with its server key. Resulting in that you have a 100% water tight cryptographically secured guarantee of that nobody have been tampering with your request somehow, and nobody has seen neither what was sent, nor what was returned.

Yet again, I want to emphasise that the code to accomplish this is not yet released, but will be released soon in the upcoming 8.4 version. If you’d like to see if 8.4 is released, you can check out this link. For the code examples, I used “Hypereval” myself, doing a simple loopback invocation to my own localhost server. You can of course use anything else as your client though, including CURL or C# code – As long as you’re able to somehow create the correct HTTP requests. And you can of course use anything as a client, and any Phosphorus Five server as a server.

As an additional bonus I have also created a GUI wrapper, allowing you to play around with Hyperlambda constructs, to test your web service code. Below is a screenshot.

When executed, a modal window will be displayed, with the same result and structure a web service invocation would yield.

PGP based server to server communication

Using PGP when two server communicates with each other has a lot of advantages, such as among other things reducing the probability of a “Man In The Middle” attack, by cryptographically signing and encrypting data sent from one server to another.

In the upcoming 8.4 release of Phosphorus Five I have made this much simpler. First of all, when you install your server, you can check of simple checkbox, and have your server’s public PGP key transferred to your configured key server. The default key server used in Phosphorus Five is “keyserver.ubuntu.com”, but this can easily be changed in your web.config file.

Secondly, when some MIME envelope is parsed, and it has been cryptographically signed, Phosphorus Five will automatically retrieve the public PGP from your configured key server, and install it into its PGP context.

Thirdly, I have created lots of meta PGP key retrieval URLs for a default Phosphorus Five installation, allowing a server to automatically communicate and send public PGP keys back and forth. For instance, if you need to securely communicate with a server using PGP cryptography, you can simply request the server’s base URL and append “/micro/pgp” to it. At which point the server’s public PGP key will be returned as ASCII armoured text. Notice, you’ll have to use “Peeples” to explicitly allow for accessing this URL if you wish for non-root accounts to be able to retrieve keys. Requesting my personal development server’s main PGP key for instance returns the following.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: BCPG C# v1.8.1.0

mQINBFsmpEUBEACPPTzy3Tg95PoQcpY6CetYbC6+1kKgn1ue9M5yMwyx/MA5JIS5
FfiQBXCzyIDTPN1xquvkAM59w0FkBfgI3ZjYojKLkx3DcEjB3acigzjxHRCvdZML
7Q6A0lWgnlV2Uw9KZmP40LHcWRSvq97eGse+wX9G278J2bLWggxVoBFXZqXnduxf
FVLMcXdA1phhk/ClMjsrPnX5tbUCXsE+wj4ZzKA81hkdg1Bq8pLeUWklzGazzUe4
lE2EJ0uekdoiMYT+7ov4H+3hf3T30naWfkR9rSb7e3owBjceJIlao3JvSfQ7jbkc
ZaOSZq1USe87PPLqvhrzNDCZqvs3EUjvXwYtGNfpDr+W/lezqTpDVenmm0g1179u
lZHpXlYkY0wqn8hY0fvjrb065Ah/FHmchzY6e4a8Lh1Iiu92pEBp6TAmdJ+AyaDF
8jsoSi+MZXKjorCmeGfiHNVCUfWL7ZMLEJOgxqpJIWxdyZ6GJOUZLs0skgCM4k5C
vXl9+ojHo+0a9pnj0xKwIsTRtipSvF+UxMgEBOHCeRC+Z9RNm9S06wzDkx2FNdi8
YC5nawSr+H1F21Ijj7nyV+B7Neb088FSnlfLn1dEcREgWhPTfu4wPy0TMzForLrQ
ULakl93CU4A57VH+8gjYuPRBk39Bz5QQhkTXVdub0Hpk4ugfXmx6LmuW6QARAQAB
tDFEdW1teSBUZXN0aW5nIEtleSBOb3QgaW4gQWN0dWFsIFVzZSA8Zm9vQGJhci5j
b20+iQIxBBABAgAbBQJbJqSCAhsDBAsJCAcGFQgCCQoLBQkFpOvFAAoJEIY6IiZv
nvOaYHIQAIvMpAdg5YsZH1cMzFFBQxMeM0hoBJottuszsuujiTyachnV8a2cZX7G
xdek2WtA/ipcuUIJJKppztjB3aNvqqUtYBhXkng0KsxWC8nctDiHNKfxsNnZ/s+4
T1Nq3cjlAH8ihWWPFCjJalwQ5nKyT+zzsVM8nfuIyjoPlZdcjbv+pJSwrsX3388L
C7J1NIxv1HYbGMT23gyPPF/S6k3mqbEbMAmphnKfMWfMDVRep8W8q4dDvPYtG3Tg
v4dwjJxhN3lzlKaSVLAauHPnVyeC44P1NMZQzXdx0Lnhd1r5ibucemxCmMrGGDvZ
/NiktmOCX9H/dykESurjBv820Xpa6r+/Z1gZ/WsaxOw3uqPyH6YgPDkdcDhisGhO
0e6yz2Sfx52VPIAw2K6EkeKLuh+MmbgEI7PZMz/j66VdsJXtqMetXCMbabWgF4V0
Sq64lcbqW0ixIFULQKEVkOCG7Jc+CaGkh1JoVcpE7YZty7wIaBPNaYYA07BhDmSc
Fs/Kb/i5AGW/cEr284k+Dx4oxKEDTmTLSgDdlNiu2Kk2KuyrwbLJPGD/iNWp7c07
Fb1e4WHyCHKaFnaWEaG6qeiLaXSyy4rHOQof0J+7/ArFc92UDtMcUWtHatIcxU0Y
vSYYzNWkLYqOghrkw1W58bGmFKa6Bzx3CzpcoQ2m350Jzhf9O2qG
=QUYz
-----END PGP PUBLIC KEY BLOCK-----

In addition, you can also list all public PGP keys a single Phosphorus Five server has by requesting the url “/micro/pgp/list”, which for my server yields the following (Hyperlambda).


83bfcbe0235f90e0a67bb865863a22266f9ef39a
  id:6F9EF39A
  algorithm:RsaGeneral
  strength:int:4096
  creation-time:date:"2018-06-17T18:11:17"
  is-encryption-key:bool:true
  is-master-key:bool:true
  is-revoked:bool:false
  version:int:4
  expires:date:"2021-06-17T18:10:18"
  user-ids
    :Dummy Testing Key Not in Actual Use <foo@bar.com>
  signed-by
    6F9EF39A:date:"2018-06-17T18:12:18"
d9d9a341717d93ce911958aeddbb618d4f2ac9a9
  id:4F2AC9A9
  algorithm:RsaGeneral
  strength:int:4096
  creation-time:date:"2018-06-17T18:07:40"
  is-encryption-key:bool:true
  is-master-key:bool:true
  is-revoked:bool:false
  version:int:4
  expires:date:"2021-06-17T18:07:03"
  user-ids
    :kgkgiygiugiugiyg iugigiug <igigiyg.ouhouh.no>
  signed-by
    4F2AC9A9:date:"2018-06-17T18:08:03"
... etc ...

… or you can query for specific keys, using a URL such as for instance “/micro/pgp/d9d9a341717d93ce911958aeddbb618d4f2ac9a9”. Which yields the following for my server.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: BCPG C# v1.8.1.0

mQINBFsmo2wBEACbEOC6qUenvvQqj2uj2RQ+eFfrcOkaQk+KKl/xoY6HDbvbQtw6
5EVx9tsftr1zzznqGT8/M2pHlJQdgaOMg8m7hzdozWkazK1L1BPEoV/r34EuLmDg
WhihMDb28Y3MBV/5Zkl9QRR5FClR2cpy/dTKc0ANwcBpqXcHzdYYviA+SoST96un
kjV85YS77WBdIEHhfBkjjDUBnjG/934y5v1txtvtBrr9MqxDJy4kgR7yHZmh4c1H
8uRkuCxWf9eJ0BH9VXhUDTDD7NTBFAdNSqH7lWfF8P8N4Z+URI6igeLET8JUZBio
5V2LvleESp9C78Sacv5j7LocqW0bL3nki0sma/yIs0OocHZxELi/or3ysW9K3KBd
z6dh/aT4TCkIh7/DuCowtUKmasfFmol/te8q2irIsBqWH32Z+7GZ1UaO9tiok2JF
BGK7+3wQcJFXnKSxppQ01KY46Nw3Pg17CgRuvMCFT+PhmGFKm8mwo2cvkq64NFU0
gjHeY4BG2iSvDLQDNfRWjA5mleUmeRBM8lYw+9Cqq3vOrWSDoLzPt9YKeKsrn3HU
SDs/IsSylIwppkpY+cKZE3zl17Eco3NwvWfWLFrr3eJxY3UsknspwQz10siEL4Z4
+Diz2oqtGPppTUBUZZOPUr1cspy61+czxEnSF+1hMVLY9tnCS12vWAQh5QARAQAB
tC1rZ2tnaXlnaXVnaXVnaXlnIGl1Z2lnaXVnIDxpZ2lnaXlnLm91aG91aC5ubz6J
AjEEEAECABsFAlsmo4MCGwMECwkIBwYVCAIJCgsFCQWk69sACgkQ3bthjU8qyalv
bxAAhwdqydcskiogAlAD1J3LarFIiWATfV5fD469xaoNq8nKLgXTDZesuCfpTY78
a5etBZDEq+UJmaxy/RfujTiDFAB3OT4mFqbToRTkTyCLXVrCivaB8gMTJS7TUNS7
/WmiV2yjSXiTPnQZbR1INgEd8JbiJ8GJbHJqHU6LPIWcSg0sH3CMuDfL6wihugFp
lpAYduDaAEhEPcK2iADcByKy6o+WUVOyivDculXkJY2migPEBhMEHoINjc9ooIep
AdyEq4HLcXPjKPCXWZp4zqX9oTCxX5q0+YC3TS6n4JKZGNMYYgMoDOq2EJ8HA5WR
RtfOq7qDQ/Vw3V2/5DBFYxwmet/U5LMBzQwCdcIXVLbBjmnclSxmbM1dg6mjAsOL
TZ3UP1s//UZB6MOzcufRaWVwFcV4sKZ+h11pwjdXER8isFfMofnze5GqtE1Lm//V
kcWOs4+gvFH3Tb4gjOXi+pIcPKxBPp8YGjSQTI2tO7N789irKATYCrfguHXHEv1V
ZjFwgIPv0NmxqHdVe0WAnjmQaaTqtLMNnUCeIV5ytM/uSKbQwkVesDZUdXgPP3Lp
ryZYKYWMeCE9epjURs3ileMfNEPOtGc27rBQHUaNHD5HG0TxqXgVqWdg1xpVJy+N
jBibMutj/fV3PzEhSUMlVN61/smJDopYBgFOCv4c06JNZbw=
=/BDf
-----END PGP PUBLIC KEY BLOCK-----

You can also of course return multiple keys at the same time, by instead passing in things that will be matched as the “identity” of the key, such as for instance “/micro/pgp/Hansen”, which will return all keys having “Hansen” somewhere within their identity.

All in all, this creates some pretty cool opportunities for secure communication, allowing for meta retrieval, having automated processes retrieve server keys, and such immediately establishing a secure and encrypted communication channel.

I will also implement more of these types of “convenience” methods and functionality before the upcoming 8.4 release, allowing you to do lots of other interesting things too. However, that was that for today 🙂

MIME Magic

In the upcoming release for Phosphorus Five, I have given a lot of attention to everything related to security, which also for obvious reasons includes how P5 creates MIME messages, and are using PGP cryptography to encrypt MIME messages. The MIME features of Phosphorus Five can be found in the project called “p5.mime”, and internally it is using MimeKit. This implies among other things that parsing and creating MIME messages in Phosphorus Five is fast as greased lightning. I have previously tested its capacity myself, and I were able to create a multipart MIME envelope containing 5,000 (small) images, encrypt the whole thing, and save it to disc – And the operation was over after 10-12 seconds on my MacBook Air from 2016, and it spent almost no memory. Decrypting the MIME envelope, and re-creating its files, also took roughly 10-12 seconds.

The sheer speed of MimeKit, and its intelligent usage of streams, hence allows us to use MIME “outside the box”. One of the things I’ll implement wrappers for probably quite soon, is the ability to invoke and handle PGP encrypted MIME web services. This has a lot of interesting advantages, such as eliminating “Man In The Middle” attacks, by expecting the web service invocation to be cryptographically signed, by a trusted client’s private PGP key. In addition to that since it’s integral to the PGP standard to actually compress the resulting encrypted content, also reducing bandwidth consumption. In addition to (of course) making a potential “Main In The Middle” completely in the dark as to what is actually transferred between the web service endpoint, and its clients. And by combining this features with lambda web services, this opens up for a whole range of interesting use cases and new approaches to solving your problems.

Notice, I want to emphasise that some of the features I describe below, are still not yet released, but will be available in Phosphorus Five as of from version 8.4 and up. If you wish to see if version 8.4 has been released, you can check out my GitHub project here. If you want to play with these features after 8.4 is released, you can use “Hypereval” and paste in the code examples I have included below …

MIME is one of those internet acronyms, and it means “Multi-purpose Internet Message Exchange”. Even though most people simply knows it indirectly through their email software, it is highly useful for a *lot* of other scenarios too, and literally a “multi purpose” format. Which I intend to illustrate with some examples at the bottom of this article. However, first let’s try to learn its basics, before we go “all Ninja” with it …

Creating a MIME envelope

To create a simple MIME envelope with Phosphorus Five is dead simple. Use for instance the [p5.mime.create] Active Event, pass in your MIME entities, and VOILA! Below is a minimalistic example illustrating this. Yet again, use “Hypereval” for the code examples in this article.

p5.mime.create
  text:plain
    content:This is your MIME content!

The above results in the following.

p5.mime.create
  result:@"Content-Type: text/plain

This is your MIME content!"

If you want to create multiple MIME messages, you can pass in multiple MIME entities, like the following illustrates.

p5.mime.create
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!

You can also provide expressions, leading to one or more nodes declaring your MIME envelopes.

.mime
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!
p5.mime.create:x:/@.mime/*

The above would result in the following.

.mime
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!
p5.mime.create
  result:@"Content-Type: text/plain

This is your MIME content!"
  result:@"Content-Type: text/plain

This is your second MIME content!"

Persisting MIME envelopes to disc

If you’d rather persist your MIME envelopes to disc, you can use [p5.mime.save]. This event requires a filename as its main argument, and expects the user to have write access to the filename you supply. Below is an example of using it.

p5.mime.save:/foo.mime
  text:plain
    content:This is your MIME content!

The above code will create a file named “foo.mime” in your server’s root folder, containing your MIME envelope. Notice, you can supply multiple MIME entities to your [p5.mime.save] event. However, this will simply persist your MIME envelopes consecutively into the same file, which is highly likely *not* what you want. Hence; “Multiparts to the rescue.”

The whole idea with MIME is that it’s actually a tree structure, or a graph object. And by creating a “multipart” MIME envelope, you can wrap multiple MIME entities into the same MIME envelope. Below is an example of creating a file on disc with a multipart containing two text MIME entities.

p5.mime.save:/foo.mime
  multipart:mixed
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

The above will result in a file resembling the following.

Content-Type: multipart/mixed; boundary="=-nqxGFPSK1IvoMUwBVqzf2g=="

--=-nqxGFPSK1IvoMUwBVqzf2g==
Content-Type: text/plain

This is your MIME content!
--=-nqxGFPSK1IvoMUwBVqzf2g==
Content-Type: text/plain

This is your OTHER MIME content!
--=-nqxGFPSK1IvoMUwBVqzf2g==--

The above is actually a tree structure, with one “multipart” containing two “text” parts. This feature of MIME allows us to embed multiple objects into a single MIME “envelope”.

Notice though that since the main argument in [p5.mime.save] is the path to your file, you cannot supply an expression leading to your MIME envelopes the way we did with [p5.mime.create]. You can though supply a path as an expression. Notice also that if you use [p5.mime.save], your MIME envelopes are serialised directly into the stream, never consuming much memory on your server, allowing you to create MIME envelopes in the gigabyte size, without exhausting your server’s resources. You can of course nest as many “multipart” MIME entities as you wish, creating entire tree structures, encrypting and cryptographically signing parts of your MIME entities, with different PGP keys too – If you wish …

Encrypting your MIME envelopes

This is where it gets really interesting. The PGP features of MimeKit, and hence Phosphorus Five, is really quite stunning. If we wanted to encrypt the above multipart for instance, we could easily do so with a couple of additional lines of code.

Background; When you installed Phosphorus Five and started it for the first time, Phosphorus Five would spend some 20-40 seconds during the initial installation process. This was the place where you had to choose a “root” password, and supply your name and email address. The reasons for this, is because it will create a PGP key pair, which is used for encrypting your “auth” file. This PGP key is your server’s PGP key pair, and can (obviously) also be used for other things too. Such as for instance encrypting your MIME envelopes. Below is an example of encrypting our “multipart” from above. To evaluate this code, you’ll have to replace the [fingerprint] value with the fingerprint for your own PGP key – Which is explained further down in this article.

p5.mime.save:/foo.mime
  multipart:mixed
    encrypt
      fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

The above would result in a file containing the following.

Content-Type: multipart/encrypted; boundary="=-s8kN2YhaHGekbhDQlp62PQ==";
	protocol="application/pgp-encrypted"

--=-s8kN2YhaHGekbhDQlp62PQ==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-s8kN2YhaHGekbhDQlp62PQ==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVbAQ/+NfMak/aAqk88ybPXT4/Z0kmKy0t4rN6M3vl5duPnsoaq
ybAF/J6JOvzJkbgnelnTBuH18mvYCvRlUqb91trUZkGEiQ69i22LJPJlilIE/lWh
E238K2WcDe7bp6Jd3aQnwRUbhxF++Ej2r7SgF46JfaYga7GPQEtTLD5vv0bupU1r
YD96IhFkn8zciDRzzUFJpBfjJqfH3/AD/m4QXFv72u/PSqpCZ4JhxDe2OWn3UjOm
D8HYUsj38zLqDr5Yu186/bKWyGN02Ma7NBJfJuTU8bdYbC+vn7Xi4vOhT94Gm2zs
TS6X77vr4MIMVyxIpbJd4ZOpgR0Od5VIZK643w2k1QPb2ZjTxrCNFlveX6AsFAbG
S/L8muQ9cnyvalgrkZ/WjhoKgXc3zJTFGoo9ssJAjr4+wV8oFB10/03WOs2K05b5
GCzGFO/gz4oNEt2hS4kiE+j4J2lRTukQivie4P66zu6o6iA+LbftypWcHn2WtCvP
T1ChBKz1q+q9OlRngZsPU3dIPrfoUNX4EoAjx1LCel1FnUnHmlIMilrpL1Cr2S4I
ReUjHzgRL/qeQlmOh4MWIk9SeHYDMNifCHOkrH+ULyuht2ySe6LdIifZDQj0X27R
fu4sqlJ5XPJLzepp/2E0f/v9S6ew+/2k2I492yhpOdp1ebCnpDkxjb0ZRy92IiTS
uwHsFWLAca9X0GDNRWOZL7oqhcOiqv20FtXJFma38onfsESuu0yhgL9sSogEFekh
rbneKuZvD4deAOObRmWghd1uq2boVysmQuc7t+20D5k9ZZNXzrcVDJfbBJiHseM+
N2liJAL+jWlZenAFYGwehvIH9OKi9kdfmsyXVxxH4Oj80sI6nrECb4FwGtv+x89F
AyoiMR+Tdj9Ja0BW0LbNen+NTxm+gLDnAnyxGUYU9focis2s0WUFqo5e480=
=EzVG
-----END PGP MESSAGE-----

--=-s8kN2YhaHGekbhDQlp62PQ==--

This is actually how your “auth” file is persisted to disc, and the above will encrypt your MIME multipart, such that only you (having access to your server’s private PGP key) can decrypt it.

Notice, you’ll need to exchange the above [fingerprint] argument to [p5.mime.save] with your server’s actual fingerprint. You can find your server’s PGP fingerprint by for instance opening up your “auth.hl” file in for instance Hyper IDE, and see the first line in it – OR better, you can use the [p5.auth.pgp.get-fingerprint] Active Event to retrieve it dynamically. Which is probably the preferred way to retrieve it, since hardcoding the fingerprint into your code, would not make your code portable. Below is an example of dynamically retrieving your server’s PGP fingerprint to accomplish the same as above.

p5.auth.pgp.get-fingerprint
p5.mime.save:/foo.md
  multipart:mixed
    encrypt
      fingerprint:x:/@p5.auth.pgp.get-fingerprint?value
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

And in fact, if you do not supply a fingerprint, but still provide an [encrypt] argument, your server’s fingerprint will be assumed, allowing you to easily encrypt your MIME envelopes with the following code.

p5.mime.create
  multipart:mixed
    encrypt
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

Cryptographically signing your MIME entities

If you want to cryptographically sign your MIME entities, this is equally easy as encrypting them. This has a lot of advantages, allowing you to create a guarantee of that the MIME envelope has not been tampered with in any ways, and originates from the party having access to the private PGP key that was used to sign the envelope. This hence gives you a cryptographic guarantee of that nobody has in any ways “tampered” with the message somehow, from the time it was created by the private PGP key that was used to sign it, to the time you received it. Below is an example.

p5.mime.create
  text:plain
    sign
      fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
        password:ThisIsNotAGoodPassword!!
    content:This is your OTHER MIME content!

The above will produce something resembling the following.

p5.mime.create
  result:@"Content-Type: multipart/signed; boundary=""=-A4YJFmKDqqYVy1NO0smuKg=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-A4YJFmKDqqYVy1NO0smuKg==
Content-Type: text/plain

This is your OTHER MIME content!
--=-A4YJFmKDqqYVy1NO0smuKg==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJhWKAAoJECVtX0VJiSVbozQP/ikZNPnNhB3w
I5A9d+h5Qn+rVJToWfRczhbpl22VymYLeKsLCrLgmSlZxGa/B8PYRwKD6FkYF6aV
nQ7QkMacnwUN429MfwhRJYSMzwRBF4q/Vdd8qaT2WEvSlEmbHg4NeVNLa/IRAnII
KNcAcS7IenILM0IBRX6CKFzPicZL9AAiNwR2x+u45+Ztzs2Rt3Z/5pnXmosO+47D
5+iTfEZJEIrAwELTSMYA/Si+9hQPTICJ5Yc+gcGSzg5JjjKHEGLodzzSxLb1bhwR
8mdUw1gktUaQscZp3kTSGUI8ZYscEjPlsGjSEwZQ9rX16guvHTEjs2qgxd2eP/4t
MIF9W22C86Bf5ffxPwS+7ldJEnaw70Q1geg1hA6K9lfHb3ckaTDdl8RJPE4KcQSv
JaE5ypr9d1LupDjtAN32V9ZSKNGwSU8gz1amx14eoQthwKRVe1kCS6j3DWGWJCbS
B4N/eLG8unkuOiCPZRz2PHGtug1dHXh4p3W7XptEcSTtDeFP0JTWawMaucS8CWNN
+7sgWm7yFxu055vhynGz7J3gGirjeLl8rJctTuXnlU9XCs/eqi/V9H2AtATeUshx
jiX90WtWbH+IaYBI3rMHWTIS7JBSpLZvuO1Kugau+TRKNFTfa21CWeSKUfgHX9QC
NAgFEWI5ytAmMufkbVXVjUcYhPTx94QTevz/BA==
=Vz1E
-----END PGP MESSAGE-----

--=-A4YJFmKDqqYVy1NO0smuKg==--
"

The PGP signature above will be verified when the MIME envelope is parsed, and if its signature is not verified, the MIME envelope will not be accepted. If as much as a single byte inside of your “text/plain” parts above has been tampered with, the signature will not be valid, and Phosphorus Five will not accept the MIME envelope. And since the signature is created with your private PGP key, and can be verified with your public PGP key, this creates a guarantee of that only the entity with access to the private PGP key was able to construct the above MIME envelope. While allowing anyone with access to your public PGP key to verify its validity.

Notice the [password] parts above. Since Phosphorus Five stores all private PGP keys synchronously encrypted, in the same format that GnuPG stores your keys – This implies that you’ll need to supply a [password] argument, in addition to your [fingerprint], to be able to sign your envelope. Yet again you can use a more dynamic approach to fetch your server’s PGP [fingerprint] and [password] such as illustrated below.

p5.auth.pgp.get-fingerprint
p5.config.get:gpg-server-keypair-password
p5.mime.create
  text:plain
    sign
      fingerprint:x:/@p5.auth.pgp.get-fingerprint?value
        password:x:/@p5.config.get/0?value
    content:This is your OTHER MIME content!

…or completely omit both the [fingerprint] and [password] parts, at which point your server’s PGP fingerprint and its password will be assumed.

p5.mime.create
  text:plain
    sign
    content:This is your OTHER MIME content!

You can of course both encrypt and sign your envelopes, using different fingerprints too if you wish. This allows you to encrypt an envelope for some recipient’s PGP key, while cryptographically signing it with your own private PGP key. This creates a guarantee of that the envelope originated from you, and that it hasn’t been tampered with – While only allowing the rightful recipient to decrypt it and see its content. And if you somehow receive a MIME envelope that has been cryptographically signed, Phosphorus Five will automatically download the public PGP key used to sign the message from a public PGP key server, if it does not have it from before. The last detail allows you to receive a cryptographically signed MIME envelope, for then to immediately start encrypting another MIME envelope to return to the sender. Immediately having established a secure communication channel. Both for authentication and encrypting the content sent back and forth.

Parsing MIME envelopes

So far we have only created MIME envelopes. Let’s look at how to parse them. This is equally easy as creating them, and can be done by for instance using the [p5.mime.parse] Active Event. Below is an example.

p5.mime.create
  text:plain
    sign
    encrypt
    content:This is your OTHER MIME content!
p5.mime.parse:x:/@p5.mime.create/0?value
  decrypt
    password:ThisIsNotAGoodPassword!!

Yet again, you can omit the [password] above, at which point your server’s PGP password will be assumed. And if you’re in the highly “paranoid” spectrum, you are of course free to change your PGP key’s AES password, which must be done before you install Phosphorus Five, by editing your web.config file. The above invocation to [p5.mime.parse] will result in something resembling the following.

p5.mime.parse
  multipart:encrypted
    signature
      thomas@gaiasoul.com:bool:true
        fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
    text:plain
      content:This is your OTHER MIME content!

Notice how the value for the above [thomas@gaiasoul.com] node is boolean true. This implies that the cryptographic signature has been verified, and that the message has not been tampered with in any ways. If it has a value of “false”, you cannot trust the message! Completely omitting all passwords and fingerprint arguments from above, to rely upon the server’s main PGP key, could be done by constructing your code like the following.

p5.mime.create
  text:plain
    sign
    encrypt
    content:This is your OTHER MIME content!
p5.mime.parse:x:/@p5.mime.create/0?value

The above works due to that if the message is encrypted, and no [password] is supplied, your server’s PGP password will be assumed, and the MIME envelope will be decrypted using your server’s PGP password, when the PGP key needed to decrypt the envelope is released from GnuPG. You can of course encrypt, sign, and decrypt multipart envelopes, nested in as many layers as you wish for that matter. Below is an example of creating and parsing a more complex MIME envelope.

p5.mime.create
  multipart:mixed
    text:plain
      sign
      encrypt
      content:Some MIME text part, this was encrypted and signed!
    text:plain
      sign
      content:Another MIME text part, this was only signed!
    multipart:mixed
      application:x-hyperlambda
        sign
        content
          foo:bar
            Hello:World
            this-was:cryptographically signed!
      text:plain
        content:Jo dude! This was not signed, nor encrypted!
p5.mime.parse:x:/@p5.mime.create/0?value

The above code will create a complex MIME tree structure with a multipart mixed as the root entity, containing two text parts, and another nested multipart, where some of your entities are signed, some are encrypted, and some are both encrypted and signed. You are of course also free to encrypt and sign your different entities with different PGP keys as you see fit. Allowing you to “triple encrypt” an envelope, with three different PGP keys, implementing arguably the “onion protocol” for web services, or emails if you wish …

Notice how we are able to directly supply “application:x-hyperlambda” above, and inject Hyperlambda directly as a node structure. This is a convenience feature, allowing you to easily transfer Hyperlambda in your MIME envelopes, and automatically have it transformed into a lambda structure on the receiving end – Assuming the entity that is parsing your MIME envelope has this feature. After executing the above, your result will resemble the following.

p5.mime.create
  result:@"Content-Type: multipart/mixed; boundary=""=-1hQmuiWvF8ISxBVeLWnprg==""

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/encrypted; boundary=""=-SS7K3W6vwN1PO8Q4J5EpCg=="";
	protocol=""application/pgp-encrypted""

--=-SS7K3W6vwN1PO8Q4J5EpCg==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-SS7K3W6vwN1PO8Q4J5EpCg==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVbARAAiyRdBIPagxCPjh+Ecnj+cogBVFvG0icGJzwX2gjfuEMP
WgOojNr66i0FiCR2fZcrWpxZ79c6Z0tYNuUq+Nr90cEYvx5pPBenfrLG8XfjTJqJ
0lQvkaSDzG0E/bc327Nue6BuYnJhfrLIVj6GZ0dj3ilzX0ly+W481i/wpPDEiM6Q
dlXeLpsyvRDg6gdC/ADbtGj85sA2v0m38vEkApn9Fapg8hKusDYj2c/Ek2OCxpyZ
NCE3xUuD27yvk7DzS7qVUeJEAujulhSlyKMNhsdtUPduBXo9Wv63IdpsfxWUvVOc
8ACO8ieMrYWHS3+paiX690MrFrayySexUwWqXRD/eLY/Cc0KJGNbw2xIhqqxdjxr
Z0Ht3morKl04IcWTosZsNhWa6bxjMfbERWafeIa20d71HHDmz//hSv9vI5xa+Cjy
/3ELrQ+ns7mrdH05qYHaUIvdPKgwvmSeSoSa0fQXn2BHFRNaA5qUsrQvP8HcsFob
vPvpemZ7f0mdrdZ20rFgirVE28DpWNjODzK6RUJYMV2zoVYhXkNK8vp0WEiHSOnz
VNSdk0pQeniOOC57m1rXSG/6TbopZh2yKsHS3v7q5mh1eK1Yxf8ZiOOe756WZsJI
7ySUzYfKDnjOHNPOSZdtyBNuetxCvVtmP0ph+ff0hYHed/edNwk0gq4al1cSofzS
whUB3QhdtUpxVT1CKPSC/KmyrREIVV0k6/99NBeDlw7fPjEzVDCG5JDcOBGmfZ9S
3Y1oWRMdaS7kC+bl1NBLEeMx29/cmzrfMNNrIQ22FVYPglrl6QPjt7vhnlr7ta5y
tSf4whrMI6jDhgP0wszgnYDCZm1ers6KsZ2VWvU/CsCDQUAZP4L3PdkxYPWZ2kji
7t2Lhsy74rt76robh5XR+fI1rPCHdKmOPR8DHs7w9lbeHDugJNRE6ZvXrIEY7SWx
Dki50f3KzJVRO0pdo8PCSAxzM5PGi15ZMsE7Hy6lykt82Fkq3dIHHXE9mwu++e5d
ElIkZvrnQxpvRCrQWwRpnUeVPgENbx+3/aA/q92NZ3u1CGJL3eIe1Sw1QqhHNhx2
TOudluBUb+Rm5S9aRHuvvlvmL3ItSVbl1FXnBID/I5IfORMq1Zq7BinfVn2+/Az1
VssVX1T6JZKR7B9LV7fhENsR0XD+1jtMt9MKtByyyAxDhMZxXBhz4uzmIBZNEJvb
Lu+uxTOAYXpEVYEuYWL4ZHVdeZvJeTTwVw4Vm6mIH+tiqX1JdO/C5jAv82/wmsl1
qHA5Kua0sLI3r9juOhNRKP59vfN6n8p74BNpwTDUsbPhGklfEuy7bhJrVZKPn7Wy
pFLONf2XKMdg8wneLVAmu8oNrcPzVr5x3ITexPcx9XbdH78UbCAZtirfW1I+pHq+
4T5ZzE0LK3MznmfLU/3tA3JDcDGAQI7SXqGOvHMeeD23kiW/jCoyhnhUQiNshIbj
FN2RfABO3lASHF9tqa5QZwLLffzch5lyKOe0wOOVqRzjAACpEkcw/70CLcmiiSLl
7gLTBiuFoPrvxt09VX2BGJoMoIAI8Y2O1iS2Xdv+7kcra5EjnFn54jeJzDJ2vCpg
vgK/hRP1SyjaPGWkuMdK2TG/q1DsbbtgoEwy1NzFsJ7WIU3n8FEueSrNhB9IUvk0
nRU4XjkpEw==
=p6eN
-----END PGP MESSAGE-----

--=-SS7K3W6vwN1PO8Q4J5EpCg==--

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/signed; boundary=""=-0sFZ7mtzSwXM4jw8kueeKA=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-0sFZ7mtzSwXM4jw8kueeKA==
Content-Type: text/plain

Another MIME text part, this was only signed!
--=-0sFZ7mtzSwXM4jw8kueeKA==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJh3iAAoJECVtX0VJiSVbIksP/A7gaFbTWVb5
sbQQqHy5MErh+ZQisB8pwHIonPdlttEK4Q+yE9rw4DVtWszO9XUhJO2xzvGX6Qqw
fkDU0M7ybKG0gMMEhwgfRc9dcSaB8MQyii5wWS9YNTXqNUmoSRS0cP6ka0h65wxh
ISQRnounQcXOIhZZywy3+YY4ZOVbv7uAmgLbBQmYTZa28wPySRlSF6U5p7h0nlJw
4Lf3+4FEtBTCjHPk6OXWpAv9cHWN7BatnoQJjFcGPks/DLy0X3UqJSrHKqOsBSa8
7Lqh/SrGBvOdkdP0Br5SNWKNGAeXrKNte3HgBNY68/IX3hAAPVd3ib2MY3oycyN0
ldzNmtVTU9ECgmz05+v35pU6unpePpKzewa2cO5koL2wnOOACOJoXisMRcXf1K/3
2/d0BP0rDGRJkc09NzLLIlBadU6iTCSo7ReQSZL9KcRCBNHkPrbLJbH3bYJBkdf+
A740RbRRY8CXswWArmAs6hg3b8ZkB8L+F2vTv3p1C311DONsugh9w1LHn9ozG+dC
E4A/zQk7zlyk0SFGm3vFfbdZf9BaL3qzMAzJzBagjO1mACHFGGYvfOcVWEcqIHMM
2U+5ACj3GjHAmGHVS3YiguF1YmLQEeImtLyW3lALrvez/bPkgQEKSXeWB6iLMT2j
MVZXtK9qsFIMQ7XHrMpTFHYewwFj/2wlqL0BXQ==
=IPqN
-----END PGP MESSAGE-----

--=-0sFZ7mtzSwXM4jw8kueeKA==--

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/mixed; boundary=""=-h2rd7/XUKjoEz8p8LHfVZQ==""

--=-h2rd7/XUKjoEz8p8LHfVZQ==
Content-Type: multipart/signed; boundary=""=-svINFQvFWbaDHisdxgmtFQ=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-svINFQvFWbaDHisdxgmtFQ==
Content-Type: application/x-hyperlambda

foo:bar
  Hello:World
  this-was:cryptographically signed!
--=-svINFQvFWbaDHisdxgmtFQ==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJh3iAAoJECVtX0VJiSVbBiEQAIWuIvQMkoa9
eMA6z0S0b5YPDtOcHK624fGh075tYz1AmXFHuND5h+2W+2ZMpoIt1Xd3+pbEaxWY
XQ/oCCb0lhkAFQGSKJzi9CNpblWECCt9E87QuSbBUGLJMmFpUUb6exWyB6uWDoMf
I3/cSTWrgbMwrnYBlsZWw3WxH1eUyID8zvPMpveIWgHoNV1REoH5/V6gNcGOEUMi
W1kE1mp4pjzAaN29S14MaqLSzM/Dz+fe/mq+tMytph1kahggs3rZSVqlvN6Io5/U
99Cg4vm38hEDbSGVjyoiNIal0VCOh/G0aHhbd/BrDR9NF7gDwuFPA0jvuxVTXssc
aPdGU4oUvMQ53eV8AFqIEyqfZVDgHNimZ8lbv/UNBqm9Et763UniZC2Xdx9iMUW7
msA3UHTYUYpr/N0uamw69jnzprf3tx00FuyV5Pn8TGkQ/rQiFgYF8QtZwi2LJYc5
wYb0bRSY5h97ynvXvJC/KbXc0e56KTDcRs6xL2/xpnHd7mmlQ8jXJhCY+TTBgIwF
uPQhafvzgAg7ZzWEeuevP7A5vEh0HZMHn0Aa8OYiMSbdVZI/d42jlfIFB8S5+DSG
+4YI4jyt4H1gN0g5RKcL2GAyOkftNySaQINb/047RoyeGvl9tpKXL7bxACZ75kXF
ud+23IkOaLNgTy0LTG2yM2pDhKHLFURVliEFCg==
=pIAf
-----END PGP MESSAGE-----

--=-svINFQvFWbaDHisdxgmtFQ==--

--=-h2rd7/XUKjoEz8p8LHfVZQ==
Content-Type: text/plain

Jo dude! This was not signed, nor encrypted!
--=-h2rd7/XUKjoEz8p8LHfVZQ==--

--=-1hQmuiWvF8ISxBVeLWnprg==--
"
p5.mime.parse
  multipart:mixed
    multipart:encrypted
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Some MIME text part, this was encrypted and signed!
    multipart:signed
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Another MIME text part, this was only signed!
      application:pgp-signature
        Content-Disposition:attachment; filename=signature.pgp
        Content-Transfer-Encoding:7bit
        content:blob:...long blob...
    multipart:mixed
      multipart:signed
        signature
          thomas@gaiasoul.com:bool:true
            fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
        application:x-hyperlambda
          content
            foo:bar
              Hello:World
              this-was:cryptographically signed!
        application:pgp-signature
          Content-Disposition:attachment; filename=signature.pgp
          Content-Transfer-Encoding:7bit
          content:blob:...long blob...
      text:plain
        content:Jo dude! This was not signed, nor encrypted!

One detail to notice in the above result, is how when you sign or encrypt a MIME entity, the entity is wrapped inside a “multipart:signed”, or “multipart:encrypted” wrapper entity. This is as expected, and necessary to correctly create your MIME envelop. To understand this process, and structure, feel free to run the following through Hypereval.

p5.mime.create
  multipart:mixed
    text:plain
      encrypt
      content:Some MIME text part, this was encrypted!
    text:plain
      sign
      content:Another MIME text part, this was only signed!
p5.mime.parse:x:/@p5.mime.create/0?value

… which of course results in the following …

p5.mime.create
  result:@"Content-Type: multipart/mixed; boundary=""=-DOjAG3V6uxgcYDTlvGj87Q==""

--=-DOjAG3V6uxgcYDTlvGj87Q==
Content-Type: multipart/encrypted; boundary=""=-vpiloBWqr+yJgOppg3UftA=="";
	protocol=""application/pgp-encrypted""

--=-vpiloBWqr+yJgOppg3UftA==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-vpiloBWqr+yJgOppg3UftA==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVblah, blah, blah ...
=rNM+
-----END PGP MESSAGE-----

--=-vpiloBWqr+yJgOppg3UftA==--

--=-DOjAG3V6uxgcYDTlvGj87Q==
Content-Type: multipart/signed; boundary=""=-li3Lt5cIlPNcSHm3d0s1eQ=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-li3Lt5cIlPNcSHm3d0s1eQ==
Content-Type: text/plain

Another MIME text part, this was only signed!
--=-li3Lt5cIlPNcSHm3d0s1eQ==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQblah, blah, blah ...
=ifqy
-----END PGP MESSAGE-----

--=-li3Lt5cIlPNcSHm3d0s1eQ==--

--=-DOjAG3V6uxgcYDTlvGj87Q==--
"
p5.mime.parse
  multipart:mixed
    multipart:encrypted
      text:plain
        content:Some MIME text part, this was encrypted!
    multipart:signed
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Another MIME text part, this was only signed!
      application:pgp-signature
        Content-Disposition:attachment; filename=signature.pgp
        Content-Transfer-Encoding:7bit
        content:blob:... some long blob ...

And since the signing key’s fingerprint is returned as a MIME envelope is being parsed, you can easily do a lookup into (for instance) a database of “trustees”, to see if the MIME envelope was signed by an endpoint you trust or not. This allows you to for instance transfer code over a web service, and execute that code at the web service endpoint, assuming the receiver trusts your PGP key’s fingerprint. However, that is the subject of another article.

MIME headers

Just like HTTP, MIME can also supply “headers”. These are pieces of “meta data” that somehow describes the envelope and entities. Basically, the way you’d supply a MIME header, is by adding it as a simple argument to your MIME entity. Below is an example.

p5.mime.create
  text:plain
    Howdy:World
    content:Foo bar content!
p5.mime.parse:x:/-/0?value

Which results in the following.

p5.mime.create
  result:@"Content-Type: text/plain
Howdy: World

Foo bar content!"
p5.mime.parse
  text:plain
    Howdy:World
    content:Foo bar content!

The idea with a MIME header, and how it is recognised as such by P5, is that it is expected to have at least one CAPITAL letter. Since all the standard MIME headers obeys by this rule, this allows you to add any MIME headers you want to – Including all the standard headers. If you’d like to supply a “Content-Transfer-Encoding” for instance, this can be accomplished with the following.

p5.mime.create
  text:plain
    Content-Transfer-Encoding:base64
    content:Foo bar content!

… resulting in the following …

p5.mime.create
  result:@"Content-Type: text/plain
Content-Transfer-Encoding: base64

Rm9vIGJhciBjb250ZW50IQ==
"

Since MimeKit and Phosphorus Five automatically handles all the default MIME headers, this allows you to use any standard MIME features you want to – In addition to providing any type of additional “meta data” to your MIME entities.

Parsing MIME envelopes directly from disc

The same way we could create a MIME envelope, and persist it to disc in one of our above examples – We can (of course) also parse a MIME entity directly from disc. For instance, in one of our above examples, we created a “foo.mime” file. To parse this file is as simple as follows.

p5.mime.load:/foo.mime

… which results in the following …

p5.mime.load
  text:plain
    content:This is your MIME content!

Of course, you can parametrise your [p5.mime.load] event the same way you can parametrise your [p5.mime.parse] events.

Attachments

The MIME standard has support for attachments. You probably knows this as “email attachments”. However, there is nothing preventing you from using it in any other types of scenarios you want to. However, before we look at file attachments, realise that instead of providing a [content] argument when creating your MIME envelopes, you can optionally instead supply a [filename] argument, pointing to a file on disc. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl

Notice, instead of providing inline [content], we declared the content as a [filename] argument. You can of course create as many MIME entities you wish, each pointing to a different [filename].

The above code will create a MIME envelope with a “Content-Disposition” MIME header containing the content of your “startup.hl” file. Its result will resemble the following.

p5.mime.create
  result:@"Content-Type: text/plain
Content-Disposition: attachment; filename=startup.hl

/*
 * Initialises Phosphorus Five.
 *
 * This is being done by evaluating all ""startup.hl"" files, for all modules found
 * in the system.
 */

...blah, blah, blah - The rest of the file ...
"

In fact, unless you explicitly add a [Content-Disposition] argument to your above [p5.mime.create] MIME entity, Phosphorus Five will automatically add one resembling your file, allowing the receiving end automatically deduct that the MIME entity is an attachment. When you parse the MIME envelope you can choose to explicitly save attachments to a folder of your choosing. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/

The above will not load any attachments into memory, but rather directly persist them to your user’s “temp” folder. This of course preserves memory, allowing you to transfer large files, and/or many files, without exhausting your server’s memory.

Notice, to avoid having filenames crash, and overwrite pre-existing files, the actual physical filename will be “randomised”. This is an automatic feature in Phosphorus Five, making the file name (by default) resemble for instance “597a148fd1984f61be9bddb3c96c4d76-startup.hl”. However, since the [p5.mime.parse] Active Event will return the filename as follows …

p5.mime.parse
  text:plain
    Content-Disposition:attachment; filename=startup.hl
    filename:startup.hl
      prefix:597a148fd1984f61be9bddb3c96c4d76-
      folder:/users/root/temp/

… you can easily deduct the actual physical filename, and its original name, as supplied when the MIME entity was created. To load the file persisted above, you can use for instance the following code.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/
load-file:~/temp/{0}{1}
  :x:/@p5.mime.parse/0/*/filename/*/prefix?value
  :x:/@p5.mime.parse/0/*/filename?value

If you don’t want the Phosphorus Five’s MIME parser to “randomise” your filenames, you can add [attachments-use-prefix] and set its value to boolean “false”. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/
  attachments-use-prefix:false

… which will result in the following …

p5.mime.create
  result:@"Content-Type: text/plain
Content-Disposition: attachment; filename=startup.hl

/*
 * Initialises Phosphorus Five.
 *
 * blah, blah, blah - Rest of file ...
 */
"
p5.mime.parse
  text:plain
    Content-Disposition:attachment; filename=startup.hl
    filename:startup.hl
      folder:/users/root/temp/

AND a file in your user’s “temp” folder named *only* “startup.hl”.

About PGP

Open PGP is a cryptography standard, allowing you to (among other things) encrypt data. It is based upon “public key cryptography” or “RSA”. You can watch a 3 minutes long video about it below.

Basically, its idea is that you can encrypt something with a public key. For then to only have the owner of the private key being able to decrypt it.

Its other part is “cryptographically signing” data. When you cryptographically sign something, you use your private key to sign it, and then anybody having access to your public key, can easily verify that the data originates from the guy having the private key – AND that it has not been tampered with since it was created.

When data is encrypted, it is also compressed (think “zipped”). This process significantly reduces the size of your data, in addition to that it becomes impossible for an adversary to see what data is being transferred.

These features combined allows you to use an inheritingly insecure communication channel (HTTP or the “World Wide Web”) to send cryptographically secured data, having a guarantee of that the data originated from who you thought it originated from, has not been read by anybody else but its recipient, and has not been tampered with by anyone since the sender created the data – In addition to that it significantly reduces the size of the data transferred, and such reduces bandwidth consumption when sending your data.

Since PGP is *integral* to Phosphorus Five, and also *blistering fast*, this allows you to do some pretty amazingly cool things with it. Phosphorus Five as of version 8.4, also have no way to export or see the private PGP key, which is by design, and hence your server has now all of a sudden the closest you can come to a 100% perfectly secured communication channel to communicate with other servers, and or encrypt data. Notice, there are no guarantees when it comes to security, but assuming there are no security holes in Phosphorus Five, or the software it relies upon, this literally makes it impossible for an adversary, including heavy duty guys such as the CIA, NSA and FSB to actually see the content transferred between your server, and whomever you choose to communicate with.

Also notice that if you want to create a backup of your server’s private and public PGP key, you can do so by finding your “.gnupg” folder beneath your Apache’s HTML folder, where you can find your PGP key inside of it. You can do such a thing using e.g. SSH. However, to minimise the risk and consequences of any (God forbids) holes in your apps, and/or Phosphorus Five – I have consciously *explicitly removed the ability to export your private PGP keys* in the 8.4 release.

With that in mind, let’s look at some use cases …

Cryptographically secured backups

Creating a really secure backup is difficult. First of all, how do you avoid allowing an adversary to see the content of your backup as you download it, or store it on some medium? Secondly, how do you know nobody have tampered with your backups, and injected some malicious piece of code or data into it, that could compromise your server as you’re restoring your backup? Well, PGP to the rescue.

p5.mime.save:~/temp/backup.mime
  application:x-hyperlambda
    encrypt
    sign
    filename:/startup.hl
micro.download.file:~/temp/backup.mime

The above code will create a backup of your “startup.hl” file, encrypt it, cryptographically sign it, and download it to your client. Meaning even if an adversary can see what is being transferred over the wire, or somehow have access to your backup – He’ll still have no idea what it contains, and the resulting backup file will also be significantly compressed. As you restore your backup, you can check its cryptographic signature, and verify that nobody have tampered with your backup. In fact, since the file is both cryptographically signed, and encrypted, this literally allows you to store your backups in a public cloud somewhere, where anyone can access it – WITHOUT anybody being able to *read* it (or tamper with it). Just remember to verify the signature as you try to restore your backup. I’ll probably at some point create convenience events wrapping this in a future release of Phosphorus Five for the record.

Cryptographically secured web services

Although this is a fairly advanced concept, I’ll go through in another article later, I still want to talk a little bit about it. The idea however, is that if you couple PGP cryptography with lambda web services, which I have written about earlier here. You can all of a sudden, highly securely, allow for execution of Hyperlambda supplied by a client to your server, and such allow only trusted clients to execute Hyperlambda in your web service endpoint. The idea is that you have some sort of list of trusted clients, which are really nothing but fingerprints for PGP keys. And only if the MIME envelope containing the Hyperlambda has been cryptographically signed by one of the clients you happen to trust, you actually go ahead and execute the code transferred.

In fact, this is such a useful scenario, I will create automatic wrappers around it, probably for the 8.4 release of Phosphorus Five.

Securely transferring multiple files

Since MIME allows you to include multiple files in one envelope, and if encrypted, also significantly compress your files – This allows you to transfer hundreds, and actually thousands of files in one simple invocation from one client to some server. The files you transfer can of course be a mixture of text files, binary files, etc – As you see fit …

I will create examples of how to do this in a later article …

Securely storing data on your server

Sometimes a piece of information is so volatile in nature, and contains extremely sensitive content, to such an extent that you need to encrypt it, and/or cryptographically sign it too. In fact, Phosphorus Five contains a very good example of such data; Its “auth.hl” file. This file contains “user settings”, in addition to passwords (in hashed/Blowfish/bcrypt format though), usernames, and other pieces of extremely sensitive information. An adversary having access to this file, can do a lot of damage, since the file might potentially contain things such as your user’s POP3 and SMTP usernames and passwords for instance. PGP to the rescue again!

If you open this file in for instance Hyper IDE, you’ll see that the file is in fact encrypted. This prevents an adversary having access to the physical file to see its content, securing your users’ data, even though a breach of your server should somehow occur.

You are of course free to encrypt any files you create locally in your Phosphorus Five installation as you see fit. With any PGP key you want, allowing your users to for instance create their own private PGP keys, never storing the GnuPG password for these keys, asking them to supply the password when decrypting the files – Eliminating the server’s “root” account from being able to read the files’ content.

AJAX upload forms

If you take a look at the [micro.widgets.upload-button], you’ll see that internally it is actually using the MIME Active Events from Phosphorus Five. This is because as I create the Ajax request on the client with JavaScript, I transfer it as a MIME message to the server, for then to parse it on the server using the MIME events from Phosphorus Five. This allows me to easily use any “advanced browser” features, while using the blistering fast MIME features of Phosphorus Five and Hyperlambda to handle the requests on the server. In addition, due to how MimeKit is being consumed internally in Phosphorus Five, it allows me to upload files in the GIGABYTE size, without exhausting the server’s memory.

MIME is more that just email

I hope I now have proven that MIME is definitely interesting for more than just emails, and in fact highly useful for a lot of different scenarios. Ranging from browser to client communication, backups, and securely PGP encrypted web services …

MIME is truly quite Magical in fact … 😉

Implementing Blowfish password storage

When it comes to security, no amounts of additional layers of security can be redundant. Even though the passwords for Phosphorus Five is already ridiculously secure, due to being PGP encrypted in a file outside of the file system accessible for the system, it doesn’t hurt to implement “slow hashing” – Even though I don’t trust it alone. Simply since it’s first of all a moving target, implying you’ll constantly have to add more workload for your own server to be secure. Secondly because when your server and your adversary’s server are 15 orders of magnitude apart in processing power, making sure your hashing algorithm is secure if it’s only based upon “slow hashing”, simply due to the nature of math, would require so much time to execute locally on your server, that it would require minutes for your users to simply be able to log in.

Regardless, I have now added Blowfish and bcrypt “slow hashing” storage of passwords in the upcoming release of Phosphorus Five. The amount of workload required to hash your passwords are set to 10 by default, but can easily be modified in your web.config. In addition each user now has his own unique salt which is used during hashing, which of course further eliminates an adversary’s ability to crack your passwords, since even two similar passwords in your password file, will still be hashed differently, due to having different salts.

In addition I have included parts of the client’s fingerprint when creating a persistent cookie on disc, if the user chooses to “Remember me” on a specific client. This further reduces the potential for credential “cookie theft”, where an adversary can pick your your cookie somehow, and use it to gain access to the system. It does however invalidate your persistent cookie if you upgrade your browser, but I think this is a small price to pay for the additional security it creates. Needless to say, but this also implies that the persistent credential cookie sent back to the client, is also not the same as the hashed password stored in your authentication file. Since the credential “ticket” sent to the client is hashed, from the Blowfish hashing result, and before it’s hashed it adds the client’s fingerprint (UserAgent and supported languages), and since it’s hashed using SHA256, which has an entropy of 2 to the power of 256, which equals 1.15e+77, which is almost the same number as the number of elementary particles in the observable universe – This makes it as far as I can see redundant to use “slow hashing” on cookie verification, and hence only when the user explicitly needs to log in, there will be a roughly 1-2 second delay while logging in, to perform the slow hashing.

Check out the code here. Using bcrypt was surprisingly easy I must confess. Literally two lines of code after having added the bcrypt package. Really nice work by the developers of bcrypt I must confess 🙂

This means that in the upcoming release, at least when it comes to authentication, Phosphorus Five literally has the same amount of security features that heavy duty things such as Linux and FreeBSD has …

… in fact, arguably better. But don’t tell Linus that … 😉