MIME Magic

In the upcoming release for Phosphorus Five, I have given a lot of attention to everything related to security, which also for obvious reasons includes how P5 creates MIME messages, and are using PGP cryptography to encrypt MIME messages. The MIME features of Phosphorus Five can be found in the project called “p5.mime”, and internally it is using MimeKit. This implies among other things that parsing and creating MIME messages in Phosphorus Five is fast as greased lightning. I have previously tested its capacity myself, and I were able to create a multipart MIME envelope containing 5,000 (small) images, encrypt the whole thing, and save it to disc – And the operation was over after 10-12 seconds on my MacBook Air from 2016, and it spent almost no memory. Decrypting the MIME envelope, and re-creating its files, also took roughly 10-12 seconds.

The sheer speed of MimeKit, and its intelligent usage of streams, hence allows us to use MIME “outside the box”. One of the things I’ll implement wrappers for probably quite soon, is the ability to invoke and handle PGP encrypted MIME web services. This has a lot of interesting advantages, such as eliminating “Man In The Middle” attacks, by expecting the web service invocation to be cryptographically signed, by a trusted client’s private PGP key. In addition to that since it’s integral to the PGP standard to actually compress the resulting encrypted content, also reducing bandwidth consumption. In addition to (of course) making a potential “Main In The Middle” completely in the dark as to what is actually transferred between the web service endpoint, and its clients. And by combining this features with lambda web services, this opens up for a whole range of interesting use cases and new approaches to solving your problems.

Notice, I want to emphasise that some of the features I describe below, are still not yet released, but will be available in Phosphorus Five as of from version 8.4 and up. If you wish to see if version 8.4 has been released, you can check out my GitHub project here. If you want to play with these features after 8.4 is released, you can use “Hypereval” and paste in the code examples I have included below …

MIME is one of those internet acronyms, and it means “Multi-purpose Internet Message Exchange”. Even though most people simply knows it indirectly through their email software, it is highly useful for a *lot* of other scenarios too, and literally a “multi purpose” format. Which I intend to illustrate with some examples at the bottom of this article. However, first let’s try to learn its basics, before we go “all Ninja” with it …

Creating a MIME envelope

To create a simple MIME envelope with Phosphorus Five is dead simple. Use for instance the [p5.mime.create] Active Event, pass in your MIME entities, and VOILA! Below is a minimalistic example illustrating this. Yet again, use “Hypereval” for the code examples in this article.

p5.mime.create
  text:plain
    content:This is your MIME content!

The above results in the following.

p5.mime.create
  result:@"Content-Type: text/plain

This is your MIME content!"

If you want to create multiple MIME messages, you can pass in multiple MIME entities, like the following illustrates.

p5.mime.create
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!

You can also provide expressions, leading to one or more nodes declaring your MIME envelopes.

.mime
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!
p5.mime.create:x:/@.mime/*

The above would result in the following.

.mime
  text:plain
    content:This is your MIME content!
  text:plain
    content:This is your second MIME content!
p5.mime.create
  result:@"Content-Type: text/plain

This is your MIME content!"
  result:@"Content-Type: text/plain

This is your second MIME content!"

Persisting MIME envelopes to disc

If you’d rather persist your MIME envelopes to disc, you can use [p5.mime.save]. This event requires a filename as its main argument, and expects the user to have write access to the filename you supply. Below is an example of using it.

p5.mime.save:/foo.mime
  text:plain
    content:This is your MIME content!

The above code will create a file named “foo.mime” in your server’s root folder, containing your MIME envelope. Notice, you can supply multiple MIME entities to your [p5.mime.save] event. However, this will simply persist your MIME envelopes consecutively into the same file, which is highly likely *not* what you want. Hence; “Multiparts to the rescue.”

The whole idea with MIME is that it’s actually a tree structure, or a graph object. And by creating a “multipart” MIME envelope, you can wrap multiple MIME entities into the same MIME envelope. Below is an example of creating a file on disc with a multipart containing two text MIME entities.

p5.mime.save:/foo.mime
  multipart:mixed
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

The above will result in a file resembling the following.

Content-Type: multipart/mixed; boundary="=-nqxGFPSK1IvoMUwBVqzf2g=="

--=-nqxGFPSK1IvoMUwBVqzf2g==
Content-Type: text/plain

This is your MIME content!
--=-nqxGFPSK1IvoMUwBVqzf2g==
Content-Type: text/plain

This is your OTHER MIME content!
--=-nqxGFPSK1IvoMUwBVqzf2g==--

The above is actually a tree structure, with one “multipart” containing two “text” parts. This feature of MIME allows us to embed multiple objects into a single MIME “envelope”.

Notice though that since the main argument in [p5.mime.save] is the path to your file, you cannot supply an expression leading to your MIME envelopes the way we did with [p5.mime.create]. You can though supply a path as an expression. Notice also that if you use [p5.mime.save], your MIME envelopes are serialised directly into the stream, never consuming much memory on your server, allowing you to create MIME envelopes in the gigabyte size, without exhausting your server’s resources. You can of course nest as many “multipart” MIME entities as you wish, creating entire tree structures, encrypting and cryptographically signing parts of your MIME entities, with different PGP keys too – If you wish …

Encrypting your MIME envelopes

This is where it gets really interesting. The PGP features of MimeKit, and hence Phosphorus Five, is really quite stunning. If we wanted to encrypt the above multipart for instance, we could easily do so with a couple of additional lines of code.

Background; When you installed Phosphorus Five and started it for the first time, Phosphorus Five would spend some 20-40 seconds during the initial installation process. This was the place where you had to choose a “root” password, and supply your name and email address. The reasons for this, is because it will create a PGP key pair, which is used for encrypting your “auth” file. This PGP key is your server’s PGP key pair, and can (obviously) also be used for other things too. Such as for instance encrypting your MIME envelopes. Below is an example of encrypting our “multipart” from above. To evaluate this code, you’ll have to replace the [fingerprint] value with the fingerprint for your own PGP key – Which is explained further down in this article.

p5.mime.save:/foo.mime
  multipart:mixed
    encrypt
      fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

The above would result in a file containing the following.

Content-Type: multipart/encrypted; boundary="=-s8kN2YhaHGekbhDQlp62PQ==";
	protocol="application/pgp-encrypted"

--=-s8kN2YhaHGekbhDQlp62PQ==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-s8kN2YhaHGekbhDQlp62PQ==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVbAQ/+NfMak/aAqk88ybPXT4/Z0kmKy0t4rN6M3vl5duPnsoaq
ybAF/J6JOvzJkbgnelnTBuH18mvYCvRlUqb91trUZkGEiQ69i22LJPJlilIE/lWh
E238K2WcDe7bp6Jd3aQnwRUbhxF++Ej2r7SgF46JfaYga7GPQEtTLD5vv0bupU1r
YD96IhFkn8zciDRzzUFJpBfjJqfH3/AD/m4QXFv72u/PSqpCZ4JhxDe2OWn3UjOm
D8HYUsj38zLqDr5Yu186/bKWyGN02Ma7NBJfJuTU8bdYbC+vn7Xi4vOhT94Gm2zs
TS6X77vr4MIMVyxIpbJd4ZOpgR0Od5VIZK643w2k1QPb2ZjTxrCNFlveX6AsFAbG
S/L8muQ9cnyvalgrkZ/WjhoKgXc3zJTFGoo9ssJAjr4+wV8oFB10/03WOs2K05b5
GCzGFO/gz4oNEt2hS4kiE+j4J2lRTukQivie4P66zu6o6iA+LbftypWcHn2WtCvP
T1ChBKz1q+q9OlRngZsPU3dIPrfoUNX4EoAjx1LCel1FnUnHmlIMilrpL1Cr2S4I
ReUjHzgRL/qeQlmOh4MWIk9SeHYDMNifCHOkrH+ULyuht2ySe6LdIifZDQj0X27R
fu4sqlJ5XPJLzepp/2E0f/v9S6ew+/2k2I492yhpOdp1ebCnpDkxjb0ZRy92IiTS
uwHsFWLAca9X0GDNRWOZL7oqhcOiqv20FtXJFma38onfsESuu0yhgL9sSogEFekh
rbneKuZvD4deAOObRmWghd1uq2boVysmQuc7t+20D5k9ZZNXzrcVDJfbBJiHseM+
N2liJAL+jWlZenAFYGwehvIH9OKi9kdfmsyXVxxH4Oj80sI6nrECb4FwGtv+x89F
AyoiMR+Tdj9Ja0BW0LbNen+NTxm+gLDnAnyxGUYU9focis2s0WUFqo5e480=
=EzVG
-----END PGP MESSAGE-----

--=-s8kN2YhaHGekbhDQlp62PQ==--

This is actually how your “auth” file is persisted to disc, and the above will encrypt your MIME multipart, such that only you (having access to your server’s private PGP key) can decrypt it.

Notice, you’ll need to exchange the above [fingerprint] argument to [p5.mime.save] with your server’s actual fingerprint. You can find your server’s PGP fingerprint by for instance opening up your “auth.hl” file in for instance Hyper IDE, and see the first line in it – OR better, you can use the [p5.auth.pgp.get-fingerprint] Active Event to retrieve it dynamically. Which is probably the preferred way to retrieve it, since hardcoding the fingerprint into your code, would not make your code portable. Below is an example of dynamically retrieving your server’s PGP fingerprint to accomplish the same as above.

p5.auth.pgp.get-fingerprint
p5.mime.save:/foo.md
  multipart:mixed
    encrypt
      fingerprint:x:/@p5.auth.pgp.get-fingerprint?value
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

And in fact, if you do not supply a fingerprint, but still provide an [encrypt] argument, your server’s fingerprint will be assumed, allowing you to easily encrypt your MIME envelopes with the following code.

p5.mime.create
  multipart:mixed
    encrypt
    text:plain
      content:This is your MIME content!
    text:plain
      content:This is your OTHER MIME content!

Cryptographically signing your MIME entities

If you want to cryptographically sign your MIME entities, this is equally easy as encrypting them. This has a lot of advantages, allowing you to create a guarantee of that the MIME envelope has not been tampered with in any ways, and originates from the party having access to the private PGP key that was used to sign the envelope. This hence gives you a cryptographic guarantee of that nobody has in any ways “tampered” with the message somehow, from the time it was created by the private PGP key that was used to sign it, to the time you received it. Below is an example.

p5.mime.create
  text:plain
    sign
      fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
        password:ThisIsNotAGoodPassword!!
    content:This is your OTHER MIME content!

The above will produce something resembling the following.

p5.mime.create
  result:@"Content-Type: multipart/signed; boundary=""=-A4YJFmKDqqYVy1NO0smuKg=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-A4YJFmKDqqYVy1NO0smuKg==
Content-Type: text/plain

This is your OTHER MIME content!
--=-A4YJFmKDqqYVy1NO0smuKg==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJhWKAAoJECVtX0VJiSVbozQP/ikZNPnNhB3w
I5A9d+h5Qn+rVJToWfRczhbpl22VymYLeKsLCrLgmSlZxGa/B8PYRwKD6FkYF6aV
nQ7QkMacnwUN429MfwhRJYSMzwRBF4q/Vdd8qaT2WEvSlEmbHg4NeVNLa/IRAnII
KNcAcS7IenILM0IBRX6CKFzPicZL9AAiNwR2x+u45+Ztzs2Rt3Z/5pnXmosO+47D
5+iTfEZJEIrAwELTSMYA/Si+9hQPTICJ5Yc+gcGSzg5JjjKHEGLodzzSxLb1bhwR
8mdUw1gktUaQscZp3kTSGUI8ZYscEjPlsGjSEwZQ9rX16guvHTEjs2qgxd2eP/4t
MIF9W22C86Bf5ffxPwS+7ldJEnaw70Q1geg1hA6K9lfHb3ckaTDdl8RJPE4KcQSv
JaE5ypr9d1LupDjtAN32V9ZSKNGwSU8gz1amx14eoQthwKRVe1kCS6j3DWGWJCbS
B4N/eLG8unkuOiCPZRz2PHGtug1dHXh4p3W7XptEcSTtDeFP0JTWawMaucS8CWNN
+7sgWm7yFxu055vhynGz7J3gGirjeLl8rJctTuXnlU9XCs/eqi/V9H2AtATeUshx
jiX90WtWbH+IaYBI3rMHWTIS7JBSpLZvuO1Kugau+TRKNFTfa21CWeSKUfgHX9QC
NAgFEWI5ytAmMufkbVXVjUcYhPTx94QTevz/BA==
=Vz1E
-----END PGP MESSAGE-----

--=-A4YJFmKDqqYVy1NO0smuKg==--
"

The PGP signature above will be verified when the MIME envelope is parsed, and if its signature is not verified, the MIME envelope will not be accepted. If as much as a single byte inside of your “text/plain” parts above has been tampered with, the signature will not be valid, and Phosphorus Five will not accept the MIME envelope. And since the signature is created with your private PGP key, and can be verified with your public PGP key, this creates a guarantee of that only the entity with access to the private PGP key was able to construct the above MIME envelope. While allowing anyone with access to your public PGP key to verify its validity.

Notice the [password] parts above. Since Phosphorus Five stores all private PGP keys synchronously encrypted, in the same format that GnuPG stores your keys – This implies that you’ll need to supply a [password] argument, in addition to your [fingerprint], to be able to sign your envelope. Yet again you can use a more dynamic approach to fetch your server’s PGP [fingerprint] and [password] such as illustrated below.

p5.auth.pgp.get-fingerprint
p5.config.get:gpg-server-keypair-password
p5.mime.create
  text:plain
    sign
      fingerprint:x:/@p5.auth.pgp.get-fingerprint?value
        password:x:/@p5.config.get/0?value
    content:This is your OTHER MIME content!

…or completely omit both the [fingerprint] and [password] parts, at which point your server’s PGP fingerprint and its password will be assumed.

p5.mime.create
  text:plain
    sign
    content:This is your OTHER MIME content!

You can of course both encrypt and sign your envelopes, using different fingerprints too if you wish. This allows you to encrypt an envelope for some recipient’s PGP key, while cryptographically signing it with your own private PGP key. This creates a guarantee of that the envelope originated from you, and that it hasn’t been tampered with – While only allowing the rightful recipient to decrypt it and see its content. And if you somehow receive a MIME envelope that has been cryptographically signed, Phosphorus Five will automatically download the public PGP key used to sign the message from a public PGP key server, if it does not have it from before. The last detail allows you to receive a cryptographically signed MIME envelope, for then to immediately start encrypting another MIME envelope to return to the sender. Immediately having established a secure communication channel. Both for authentication and encrypting the content sent back and forth.

Parsing MIME envelopes

So far we have only created MIME envelopes. Let’s look at how to parse them. This is equally easy as creating them, and can be done by for instance using the [p5.mime.parse] Active Event. Below is an example.

p5.mime.create
  text:plain
    sign
    encrypt
    content:This is your OTHER MIME content!
p5.mime.parse:x:/@p5.mime.create/0?value
  decrypt
    password:ThisIsNotAGoodPassword!!

Yet again, you can omit the [password] above, at which point your server’s PGP password will be assumed. And if you’re in the highly “paranoid” spectrum, you are of course free to change your PGP key’s AES password, which must be done before you install Phosphorus Five, by editing your web.config file. The above invocation to [p5.mime.parse] will result in something resembling the following.

p5.mime.parse
  multipart:encrypted
    signature
      thomas@gaiasoul.com:bool:true
        fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
    text:plain
      content:This is your OTHER MIME content!

Notice how the value for the above [thomas@gaiasoul.com] node is boolean true. This implies that the cryptographic signature has been verified, and that the message has not been tampered with in any ways. If it has a value of “false”, you cannot trust the message! Completely omitting all passwords and fingerprint arguments from above, to rely upon the server’s main PGP key, could be done by constructing your code like the following.

p5.mime.create
  text:plain
    sign
    encrypt
    content:This is your OTHER MIME content!
p5.mime.parse:x:/@p5.mime.create/0?value

The above works due to that if the message is encrypted, and no [password] is supplied, your server’s PGP password will be assumed, and the MIME envelope will be decrypted using your server’s PGP password, when the PGP key needed to decrypt the envelope is released from GnuPG. You can of course encrypt, sign, and decrypt multipart envelopes, nested in as many layers as you wish for that matter. Below is an example of creating and parsing a more complex MIME envelope.

p5.mime.create
  multipart:mixed
    text:plain
      sign
      encrypt
      content:Some MIME text part, this was encrypted and signed!
    text:plain
      sign
      content:Another MIME text part, this was only signed!
    multipart:mixed
      application:x-hyperlambda
        sign
        content
          foo:bar
            Hello:World
            this-was:cryptographically signed!
      text:plain
        content:Jo dude! This was not signed, nor encrypted!
p5.mime.parse:x:/@p5.mime.create/0?value

The above code will create a complex MIME tree structure with a multipart mixed as the root entity, containing two text parts, and another nested multipart, where some of your entities are signed, some are encrypted, and some are both encrypted and signed. You are of course also free to encrypt and sign your different entities with different PGP keys as you see fit. Allowing you to “triple encrypt” an envelope, with three different PGP keys, implementing arguably the “onion protocol” for web services, or emails if you wish …

Notice how we are able to directly supply “application:x-hyperlambda” above, and inject Hyperlambda directly as a node structure. This is a convenience feature, allowing you to easily transfer Hyperlambda in your MIME envelopes, and automatically have it transformed into a lambda structure on the receiving end – Assuming the entity that is parsing your MIME envelope has this feature. After executing the above, your result will resemble the following.

p5.mime.create
  result:@"Content-Type: multipart/mixed; boundary=""=-1hQmuiWvF8ISxBVeLWnprg==""

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/encrypted; boundary=""=-SS7K3W6vwN1PO8Q4J5EpCg=="";
	protocol=""application/pgp-encrypted""

--=-SS7K3W6vwN1PO8Q4J5EpCg==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-SS7K3W6vwN1PO8Q4J5EpCg==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVbARAAiyRdBIPagxCPjh+Ecnj+cogBVFvG0icGJzwX2gjfuEMP
WgOojNr66i0FiCR2fZcrWpxZ79c6Z0tYNuUq+Nr90cEYvx5pPBenfrLG8XfjTJqJ
0lQvkaSDzG0E/bc327Nue6BuYnJhfrLIVj6GZ0dj3ilzX0ly+W481i/wpPDEiM6Q
dlXeLpsyvRDg6gdC/ADbtGj85sA2v0m38vEkApn9Fapg8hKusDYj2c/Ek2OCxpyZ
NCE3xUuD27yvk7DzS7qVUeJEAujulhSlyKMNhsdtUPduBXo9Wv63IdpsfxWUvVOc
8ACO8ieMrYWHS3+paiX690MrFrayySexUwWqXRD/eLY/Cc0KJGNbw2xIhqqxdjxr
Z0Ht3morKl04IcWTosZsNhWa6bxjMfbERWafeIa20d71HHDmz//hSv9vI5xa+Cjy
/3ELrQ+ns7mrdH05qYHaUIvdPKgwvmSeSoSa0fQXn2BHFRNaA5qUsrQvP8HcsFob
vPvpemZ7f0mdrdZ20rFgirVE28DpWNjODzK6RUJYMV2zoVYhXkNK8vp0WEiHSOnz
VNSdk0pQeniOOC57m1rXSG/6TbopZh2yKsHS3v7q5mh1eK1Yxf8ZiOOe756WZsJI
7ySUzYfKDnjOHNPOSZdtyBNuetxCvVtmP0ph+ff0hYHed/edNwk0gq4al1cSofzS
whUB3QhdtUpxVT1CKPSC/KmyrREIVV0k6/99NBeDlw7fPjEzVDCG5JDcOBGmfZ9S
3Y1oWRMdaS7kC+bl1NBLEeMx29/cmzrfMNNrIQ22FVYPglrl6QPjt7vhnlr7ta5y
tSf4whrMI6jDhgP0wszgnYDCZm1ers6KsZ2VWvU/CsCDQUAZP4L3PdkxYPWZ2kji
7t2Lhsy74rt76robh5XR+fI1rPCHdKmOPR8DHs7w9lbeHDugJNRE6ZvXrIEY7SWx
Dki50f3KzJVRO0pdo8PCSAxzM5PGi15ZMsE7Hy6lykt82Fkq3dIHHXE9mwu++e5d
ElIkZvrnQxpvRCrQWwRpnUeVPgENbx+3/aA/q92NZ3u1CGJL3eIe1Sw1QqhHNhx2
TOudluBUb+Rm5S9aRHuvvlvmL3ItSVbl1FXnBID/I5IfORMq1Zq7BinfVn2+/Az1
VssVX1T6JZKR7B9LV7fhENsR0XD+1jtMt9MKtByyyAxDhMZxXBhz4uzmIBZNEJvb
Lu+uxTOAYXpEVYEuYWL4ZHVdeZvJeTTwVw4Vm6mIH+tiqX1JdO/C5jAv82/wmsl1
qHA5Kua0sLI3r9juOhNRKP59vfN6n8p74BNpwTDUsbPhGklfEuy7bhJrVZKPn7Wy
pFLONf2XKMdg8wneLVAmu8oNrcPzVr5x3ITexPcx9XbdH78UbCAZtirfW1I+pHq+
4T5ZzE0LK3MznmfLU/3tA3JDcDGAQI7SXqGOvHMeeD23kiW/jCoyhnhUQiNshIbj
FN2RfABO3lASHF9tqa5QZwLLffzch5lyKOe0wOOVqRzjAACpEkcw/70CLcmiiSLl
7gLTBiuFoPrvxt09VX2BGJoMoIAI8Y2O1iS2Xdv+7kcra5EjnFn54jeJzDJ2vCpg
vgK/hRP1SyjaPGWkuMdK2TG/q1DsbbtgoEwy1NzFsJ7WIU3n8FEueSrNhB9IUvk0
nRU4XjkpEw==
=p6eN
-----END PGP MESSAGE-----

--=-SS7K3W6vwN1PO8Q4J5EpCg==--

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/signed; boundary=""=-0sFZ7mtzSwXM4jw8kueeKA=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-0sFZ7mtzSwXM4jw8kueeKA==
Content-Type: text/plain

Another MIME text part, this was only signed!
--=-0sFZ7mtzSwXM4jw8kueeKA==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJh3iAAoJECVtX0VJiSVbIksP/A7gaFbTWVb5
sbQQqHy5MErh+ZQisB8pwHIonPdlttEK4Q+yE9rw4DVtWszO9XUhJO2xzvGX6Qqw
fkDU0M7ybKG0gMMEhwgfRc9dcSaB8MQyii5wWS9YNTXqNUmoSRS0cP6ka0h65wxh
ISQRnounQcXOIhZZywy3+YY4ZOVbv7uAmgLbBQmYTZa28wPySRlSF6U5p7h0nlJw
4Lf3+4FEtBTCjHPk6OXWpAv9cHWN7BatnoQJjFcGPks/DLy0X3UqJSrHKqOsBSa8
7Lqh/SrGBvOdkdP0Br5SNWKNGAeXrKNte3HgBNY68/IX3hAAPVd3ib2MY3oycyN0
ldzNmtVTU9ECgmz05+v35pU6unpePpKzewa2cO5koL2wnOOACOJoXisMRcXf1K/3
2/d0BP0rDGRJkc09NzLLIlBadU6iTCSo7ReQSZL9KcRCBNHkPrbLJbH3bYJBkdf+
A740RbRRY8CXswWArmAs6hg3b8ZkB8L+F2vTv3p1C311DONsugh9w1LHn9ozG+dC
E4A/zQk7zlyk0SFGm3vFfbdZf9BaL3qzMAzJzBagjO1mACHFGGYvfOcVWEcqIHMM
2U+5ACj3GjHAmGHVS3YiguF1YmLQEeImtLyW3lALrvez/bPkgQEKSXeWB6iLMT2j
MVZXtK9qsFIMQ7XHrMpTFHYewwFj/2wlqL0BXQ==
=IPqN
-----END PGP MESSAGE-----

--=-0sFZ7mtzSwXM4jw8kueeKA==--

--=-1hQmuiWvF8ISxBVeLWnprg==
Content-Type: multipart/mixed; boundary=""=-h2rd7/XUKjoEz8p8LHfVZQ==""

--=-h2rd7/XUKjoEz8p8LHfVZQ==
Content-Type: multipart/signed; boundary=""=-svINFQvFWbaDHisdxgmtFQ=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-svINFQvFWbaDHisdxgmtFQ==
Content-Type: application/x-hyperlambda

foo:bar
  Hello:World
  this-was:cryptographically signed!
--=-svINFQvFWbaDHisdxgmtFQ==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQIcBAEBCAAGBQJbJh3iAAoJECVtX0VJiSVbBiEQAIWuIvQMkoa9
eMA6z0S0b5YPDtOcHK624fGh075tYz1AmXFHuND5h+2W+2ZMpoIt1Xd3+pbEaxWY
XQ/oCCb0lhkAFQGSKJzi9CNpblWECCt9E87QuSbBUGLJMmFpUUb6exWyB6uWDoMf
I3/cSTWrgbMwrnYBlsZWw3WxH1eUyID8zvPMpveIWgHoNV1REoH5/V6gNcGOEUMi
W1kE1mp4pjzAaN29S14MaqLSzM/Dz+fe/mq+tMytph1kahggs3rZSVqlvN6Io5/U
99Cg4vm38hEDbSGVjyoiNIal0VCOh/G0aHhbd/BrDR9NF7gDwuFPA0jvuxVTXssc
aPdGU4oUvMQ53eV8AFqIEyqfZVDgHNimZ8lbv/UNBqm9Et763UniZC2Xdx9iMUW7
msA3UHTYUYpr/N0uamw69jnzprf3tx00FuyV5Pn8TGkQ/rQiFgYF8QtZwi2LJYc5
wYb0bRSY5h97ynvXvJC/KbXc0e56KTDcRs6xL2/xpnHd7mmlQ8jXJhCY+TTBgIwF
uPQhafvzgAg7ZzWEeuevP7A5vEh0HZMHn0Aa8OYiMSbdVZI/d42jlfIFB8S5+DSG
+4YI4jyt4H1gN0g5RKcL2GAyOkftNySaQINb/047RoyeGvl9tpKXL7bxACZ75kXF
ud+23IkOaLNgTy0LTG2yM2pDhKHLFURVliEFCg==
=pIAf
-----END PGP MESSAGE-----

--=-svINFQvFWbaDHisdxgmtFQ==--

--=-h2rd7/XUKjoEz8p8LHfVZQ==
Content-Type: text/plain

Jo dude! This was not signed, nor encrypted!
--=-h2rd7/XUKjoEz8p8LHfVZQ==--

--=-1hQmuiWvF8ISxBVeLWnprg==--
"
p5.mime.parse
  multipart:mixed
    multipart:encrypted
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Some MIME text part, this was encrypted and signed!
    multipart:signed
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Another MIME text part, this was only signed!
      application:pgp-signature
        Content-Disposition:attachment; filename=signature.pgp
        Content-Transfer-Encoding:7bit
        content:blob:...long blob...
    multipart:mixed
      multipart:signed
        signature
          thomas@gaiasoul.com:bool:true
            fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
        application:x-hyperlambda
          content
            foo:bar
              Hello:World
              this-was:cryptographically signed!
        application:pgp-signature
          Content-Disposition:attachment; filename=signature.pgp
          Content-Transfer-Encoding:7bit
          content:blob:...long blob...
      text:plain
        content:Jo dude! This was not signed, nor encrypted!

One detail to notice in the above result, is how when you sign or encrypt a MIME entity, the entity is wrapped inside a “multipart:signed”, or “multipart:encrypted” wrapper entity. This is as expected, and necessary to correctly create your MIME envelop. To understand this process, and structure, feel free to run the following through Hypereval.

p5.mime.create
  multipart:mixed
    text:plain
      encrypt
      content:Some MIME text part, this was encrypted!
    text:plain
      sign
      content:Another MIME text part, this was only signed!
p5.mime.parse:x:/@p5.mime.create/0?value

… which of course results in the following …

p5.mime.create
  result:@"Content-Type: multipart/mixed; boundary=""=-DOjAG3V6uxgcYDTlvGj87Q==""

--=-DOjAG3V6uxgcYDTlvGj87Q==
Content-Type: multipart/encrypted; boundary=""=-vpiloBWqr+yJgOppg3UftA=="";
	protocol=""application/pgp-encrypted""

--=-vpiloBWqr+yJgOppg3UftA==
Content-Type: application/pgp-encrypted
Content-Disposition: attachment
Content-Transfer-Encoding: 7bit

Version: 1

--=-vpiloBWqr+yJgOppg3UftA==
Content-Type: application/octet-stream
Content-Disposition: attachment

-----BEGIN PGP MESSAGE-----
Version: 

hQIMAyVtX0VJiSVblah, blah, blah ...
=rNM+
-----END PGP MESSAGE-----

--=-vpiloBWqr+yJgOppg3UftA==--

--=-DOjAG3V6uxgcYDTlvGj87Q==
Content-Type: multipart/signed; boundary=""=-li3Lt5cIlPNcSHm3d0s1eQ=="";
	protocol=""application/pgp-signature""; micalg=pgp-sha256

--=-li3Lt5cIlPNcSHm3d0s1eQ==
Content-Type: text/plain

Another MIME text part, this was only signed!
--=-li3Lt5cIlPNcSHm3d0s1eQ==
Content-Type: application/pgp-signature; name=signature.pgp
Content-Disposition: attachment; filename=signature.pgp
Content-Transfer-Encoding: 7bit

-----BEGIN PGP MESSAGE-----
Version: 

owJ4nAEfAuD9iQblah, blah, blah ...
=ifqy
-----END PGP MESSAGE-----

--=-li3Lt5cIlPNcSHm3d0s1eQ==--

--=-DOjAG3V6uxgcYDTlvGj87Q==--
"
p5.mime.parse
  multipart:mixed
    multipart:encrypted
      text:plain
        content:Some MIME text part, this was encrypted!
    multipart:signed
      signature
        thomas@gaiasoul.com:bool:true
          fingerprint:E43BB8C3A3B9E8DE980051A3256D5F454989255B
      text:plain
        content:Another MIME text part, this was only signed!
      application:pgp-signature
        Content-Disposition:attachment; filename=signature.pgp
        Content-Transfer-Encoding:7bit
        content:blob:... some long blob ...

And since the signing key’s fingerprint is returned as a MIME envelope is being parsed, you can easily do a lookup into (for instance) a database of “trustees”, to see if the MIME envelope was signed by an endpoint you trust or not. This allows you to for instance transfer code over a web service, and execute that code at the web service endpoint, assuming the receiver trusts your PGP key’s fingerprint. However, that is the subject of another article.

MIME headers

Just like HTTP, MIME can also supply “headers”. These are pieces of “meta data” that somehow describes the envelope and entities. Basically, the way you’d supply a MIME header, is by adding it as a simple argument to your MIME entity. Below is an example.

p5.mime.create
  text:plain
    Howdy:World
    content:Foo bar content!
p5.mime.parse:x:/-/0?value

Which results in the following.

p5.mime.create
  result:@"Content-Type: text/plain
Howdy: World

Foo bar content!"
p5.mime.parse
  text:plain
    Howdy:World
    content:Foo bar content!

The idea with a MIME header, and how it is recognised as such by P5, is that it is expected to have at least one CAPITAL letter. Since all the standard MIME headers obeys by this rule, this allows you to add any MIME headers you want to – Including all the standard headers. If you’d like to supply a “Content-Transfer-Encoding” for instance, this can be accomplished with the following.

p5.mime.create
  text:plain
    Content-Transfer-Encoding:base64
    content:Foo bar content!

… resulting in the following …

p5.mime.create
  result:@"Content-Type: text/plain
Content-Transfer-Encoding: base64

Rm9vIGJhciBjb250ZW50IQ==
"

Since MimeKit and Phosphorus Five automatically handles all the default MIME headers, this allows you to use any standard MIME features you want to – In addition to providing any type of additional “meta data” to your MIME entities.

Parsing MIME envelopes directly from disc

The same way we could create a MIME envelope, and persist it to disc in one of our above examples – We can (of course) also parse a MIME entity directly from disc. For instance, in one of our above examples, we created a “foo.mime” file. To parse this file is as simple as follows.

p5.mime.load:/foo.mime

… which results in the following …

p5.mime.load
  text:plain
    content:This is your MIME content!

Of course, you can parametrise your [p5.mime.load] event the same way you can parametrise your [p5.mime.parse] events.

Attachments

The MIME standard has support for attachments. You probably knows this as “email attachments”. However, there is nothing preventing you from using it in any other types of scenarios you want to. However, before we look at file attachments, realise that instead of providing a [content] argument when creating your MIME envelopes, you can optionally instead supply a [filename] argument, pointing to a file on disc. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl

Notice, instead of providing inline [content], we declared the content as a [filename] argument. You can of course create as many MIME entities you wish, each pointing to a different [filename].

The above code will create a MIME envelope with a “Content-Disposition” MIME header containing the content of your “startup.hl” file. Its result will resemble the following.

p5.mime.create
  result:@"Content-Type: text/plain
Content-Disposition: attachment; filename=startup.hl

/*
 * Initialises Phosphorus Five.
 *
 * This is being done by evaluating all ""startup.hl"" files, for all modules found
 * in the system.
 */

...blah, blah, blah - The rest of the file ...
"

In fact, unless you explicitly add a [Content-Disposition] argument to your above [p5.mime.create] MIME entity, Phosphorus Five will automatically add one resembling your file, allowing the receiving end automatically deduct that the MIME entity is an attachment. When you parse the MIME envelope you can choose to explicitly save attachments to a folder of your choosing. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/

The above will not load any attachments into memory, but rather directly persist them to your user’s “temp” folder. This of course preserves memory, allowing you to transfer large files, and/or many files, without exhausting your server’s memory.

Notice, to avoid having filenames crash, and overwrite pre-existing files, the actual physical filename will be “randomised”. This is an automatic feature in Phosphorus Five, making the file name (by default) resemble for instance “597a148fd1984f61be9bddb3c96c4d76-startup.hl”. However, since the [p5.mime.parse] Active Event will return the filename as follows …

p5.mime.parse
  text:plain
    Content-Disposition:attachment; filename=startup.hl
    filename:startup.hl
      prefix:597a148fd1984f61be9bddb3c96c4d76-
      folder:/users/root/temp/

… you can easily deduct the actual physical filename, and its original name, as supplied when the MIME entity was created. To load the file persisted above, you can use for instance the following code.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/
load-file:~/temp/{0}{1}
  :x:/@p5.mime.parse/0/*/filename/*/prefix?value
  :x:/@p5.mime.parse/0/*/filename?value

If you don’t want the Phosphorus Five’s MIME parser to “randomise” your filenames, you can add [attachments-use-prefix] and set its value to boolean “false”. Below is an example.

p5.mime.create
  text:plain
    filename:/startup.hl
p5.mime.parse:x:/-/0?value
  attachment-folder:~/temp/
  attachments-use-prefix:false

… which will result in the following …

p5.mime.create
  result:@"Content-Type: text/plain
Content-Disposition: attachment; filename=startup.hl

/*
 * Initialises Phosphorus Five.
 *
 * blah, blah, blah - Rest of file ...
 */
"
p5.mime.parse
  text:plain
    Content-Disposition:attachment; filename=startup.hl
    filename:startup.hl
      folder:/users/root/temp/

AND a file in your user’s “temp” folder named *only* “startup.hl”.

About PGP

Open PGP is a cryptography standard, allowing you to (among other things) encrypt data. It is based upon “public key cryptography” or “RSA”. You can watch a 3 minutes long video about it below.

Basically, its idea is that you can encrypt something with a public key. For then to only have the owner of the private key being able to decrypt it.

Its other part is “cryptographically signing” data. When you cryptographically sign something, you use your private key to sign it, and then anybody having access to your public key, can easily verify that the data originates from the guy having the private key – AND that it has not been tampered with since it was created.

When data is encrypted, it is also compressed (think “zipped”). This process significantly reduces the size of your data, in addition to that it becomes impossible for an adversary to see what data is being transferred.

These features combined allows you to use an inheritingly insecure communication channel (HTTP or the “World Wide Web”) to send cryptographically secured data, having a guarantee of that the data originated from who you thought it originated from, has not been read by anybody else but its recipient, and has not been tampered with by anyone since the sender created the data – In addition to that it significantly reduces the size of the data transferred, and such reduces bandwidth consumption when sending your data.

Since PGP is *integral* to Phosphorus Five, and also *blistering fast*, this allows you to do some pretty amazingly cool things with it. Phosphorus Five as of version 8.4, also have no way to export or see the private PGP key, which is by design, and hence your server has now all of a sudden the closest you can come to a 100% perfectly secured communication channel to communicate with other servers, and or encrypt data. Notice, there are no guarantees when it comes to security, but assuming there are no security holes in Phosphorus Five, or the software it relies upon, this literally makes it impossible for an adversary, including heavy duty guys such as the CIA, NSA and FSB to actually see the content transferred between your server, and whomever you choose to communicate with.

Also notice that if you want to create a backup of your server’s private and public PGP key, you can do so by finding your “.gnupg” folder beneath your Apache’s HTML folder, where you can find your PGP key inside of it. You can do such a thing using e.g. SSH. However, to minimise the risk and consequences of any (God forbids) holes in your apps, and/or Phosphorus Five – I have consciously *explicitly removed the ability to export your private PGP keys* in the 8.4 release.

With that in mind, let’s look at some use cases …

Cryptographically secured backups

Creating a really secure backup is difficult. First of all, how do you avoid allowing an adversary to see the content of your backup as you download it, or store it on some medium? Secondly, how do you know nobody have tampered with your backups, and injected some malicious piece of code or data into it, that could compromise your server as you’re restoring your backup? Well, PGP to the rescue.

p5.mime.save:~/temp/backup.mime
  application:x-hyperlambda
    encrypt
    sign
    filename:/startup.hl
micro.download.file:~/temp/backup.mime

The above code will create a backup of your “startup.hl” file, encrypt it, cryptographically sign it, and download it to your client. Meaning even if an adversary can see what is being transferred over the wire, or somehow have access to your backup – He’ll still have no idea what it contains, and the resulting backup file will also be significantly compressed. As you restore your backup, you can check its cryptographic signature, and verify that nobody have tampered with your backup. In fact, since the file is both cryptographically signed, and encrypted, this literally allows you to store your backups in a public cloud somewhere, where anyone can access it – WITHOUT anybody being able to *read* it (or tamper with it). Just remember to verify the signature as you try to restore your backup. I’ll probably at some point create convenience events wrapping this in a future release of Phosphorus Five for the record.

Cryptographically secured web services

Although this is a fairly advanced concept, I’ll go through in another article later, I still want to talk a little bit about it. The idea however, is that if you couple PGP cryptography with lambda web services, which I have written about earlier here. You can all of a sudden, highly securely, allow for execution of Hyperlambda supplied by a client to your server, and such allow only trusted clients to execute Hyperlambda in your web service endpoint. The idea is that you have some sort of list of trusted clients, which are really nothing but fingerprints for PGP keys. And only if the MIME envelope containing the Hyperlambda has been cryptographically signed by one of the clients you happen to trust, you actually go ahead and execute the code transferred.

In fact, this is such a useful scenario, I will create automatic wrappers around it, probably for the 8.4 release of Phosphorus Five.

Securely transferring multiple files

Since MIME allows you to include multiple files in one envelope, and if encrypted, also significantly compress your files – This allows you to transfer hundreds, and actually thousands of files in one simple invocation from one client to some server. The files you transfer can of course be a mixture of text files, binary files, etc – As you see fit …

I will create examples of how to do this in a later article …

Securely storing data on your server

Sometimes a piece of information is so volatile in nature, and contains extremely sensitive content, to such an extent that you need to encrypt it, and/or cryptographically sign it too. In fact, Phosphorus Five contains a very good example of such data; Its “auth.hl” file. This file contains “user settings”, in addition to passwords (in hashed/Blowfish/bcrypt format though), usernames, and other pieces of extremely sensitive information. An adversary having access to this file, can do a lot of damage, since the file might potentially contain things such as your user’s POP3 and SMTP usernames and passwords for instance. PGP to the rescue again!

If you open this file in for instance Hyper IDE, you’ll see that the file is in fact encrypted. This prevents an adversary having access to the physical file to see its content, securing your users’ data, even though a breach of your server should somehow occur.

You are of course free to encrypt any files you create locally in your Phosphorus Five installation as you see fit. With any PGP key you want, allowing your users to for instance create their own private PGP keys, never storing the GnuPG password for these keys, asking them to supply the password when decrypting the files – Eliminating the server’s “root” account from being able to read the files’ content.

AJAX upload forms

If you take a look at the [micro.widgets.upload-button], you’ll see that internally it is actually using the MIME Active Events from Phosphorus Five. This is because as I create the Ajax request on the client with JavaScript, I transfer it as a MIME message to the server, for then to parse it on the server using the MIME events from Phosphorus Five. This allows me to easily use any “advanced browser” features, while using the blistering fast MIME features of Phosphorus Five and Hyperlambda to handle the requests on the server. In addition, due to how MimeKit is being consumed internally in Phosphorus Five, it allows me to upload files in the GIGABYTE size, without exhausting the server’s memory.

MIME is more that just email

I hope I now have proven that MIME is definitely interesting for more than just emails, and in fact highly useful for a lot of different scenarios. Ranging from browser to client communication, backups, and securely PGP encrypted web services …

MIME is truly quite Magical in fact … 😉

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.