Create an account


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Tut] Python Base64 – String Encoding and Decoding [+Video]

#1
Python Base64 – String Encoding and Decoding [+Video]

<div><figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<iframe loading="lazy" title="Python Base64 - Mastering String Encoding and Decoding" width="780" height="439" src="https://www.youtube.com/embed/RfKAVNGg9Kc?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
</figure>
<h2>A Short Guide to Base64’s History and Purpose</h2>
<p class="has-global-color-8-background-color has-background">Base64 is a system of binary-to-text transcoding schemas, which enable the bidirectional transformation of various binary and non-binary content to plain text and back. </p>
<p>Compared to binary content, storage and transfer of textual content over the network is significantly simplified and opens many possibilities for flexible data exchange and processing between different, heterogeneous information systems. </p>
<p>One of the <strong>key benefits</strong> of Base64 is the possibility of <strong>data transfer over e-mail attachments</strong>. </p>
<p>Why is this important? </p>
<p><img src="https://s.w.org/images/core/emoji/13.1.0/72x72/1f4a1.png" alt="?" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Well, as e-mail is one of the oldest and most used communication technologies, dating back to 1971, it is a very common way of conveying information between a sender and any number of receivers. The information is most often delivered as a readable text message, but it can also carry an attachment in a binary form. </p>
<p>E-mail servers and clients support transfers of attached binary content by converting it to plain text before sending and back to the original binary content after receiving. </p>
<p>Of course, there is some overhead in terms of time needed for conversion of the content, but it lends to the flexibility of unrestricted content exchange. </p>
<p>Some other uses of Base64 are the <strong>storage of binary content</strong> in information systems, such as <strong>databases</strong>. </p>
<p>Base64 should not be mistaken for a <strong>cryptographic algorithm</strong>, because it does <strong><em>not </em></strong>encrypt the data. It just converts the data to plain text and the transcoding schema is very well known, so there is no secrecy involved.</p>
<h2>Base64 Memory Addressing Schema</h2>
<p>Before we move on, I’d like to share a couple of thoughts on memory addressing schema. </p>
<p>When a computer stores and handles the data in its memory, namely the Random Access Memory (RAM), it has to use an addressing scheme. </p>
<p>In a common computer architecture model, the memory locations are accessed by their addresses. </p>
<p>Although it is not the only approach, we will take a look at how the byte addressing scheme works because it is the best approximation for our explanation of the Base64 mechanism. </p>
<p>When a memory location is addressed, a specific amount of data, i.e. 1 byte (equals to eight bits) is read from it. This one byte is used as the smallest addressable amount of data available. </p>
<p>If in some imaginary case we would have to store less than one byte of data, let’s say, only three bits, such as 101, that data would get padded with zeroes and would look like 0000 0101 (don’t mind the space, it is here just to make the example more readable). </p>
<p>In a described case, we say that the addressing scheme used is <strong><em>byte-addressing</em></strong>. </p>
<p>In some other cases, where the addressing scheme would use more than one byte (but always the multiple), like four bytes (32 bits = 1 word), each data shorter than 4 bytes would get padded to the full length of 4 bytes. </p>
<p>Accordingly, if the data we are handing is longer than four bytes, it would get split between more than one memory location by the formula: <code>ceil(data_len / mem_loc_len)</code>, where </p>
<ul>
<li><code><a href="https://blog.finxter.com/python-math-functions/" data-type="post" data-id="69128" target="_blank" rel="noreferrer noopener">ceil()</a></code> is a mathematical function for up-rounding to the nearest higher integer, </li>
<li><code>date_len</code> is the length of our data in bits (bytes) and </li>
<li><code>mem_loc_len</code> is the length of our memory location in bits (bytes). </li>
</ul>
<p>In other words, if our data is 33 bits long and our memory location is 32 bits long, our data would occupy two memory locations because <code>ceil(33/32) = 2</code>.</p>
<h2>Base64 Transcoding Table</h2>
<p>Base64 schema uses 64 characters (hence the name), which can be encoded with only six bits, since 2^6 = 64. </p>
<p>Before we go more specific about the transcoding mechanism, we should construct our transcoding table of symbols. The standard way of doing so is defined by <a rel="noreferrer noopener" href="https://datatracker.ietf.org/doc/html/rfc4648" data-type="URL" data-id="https://datatracker.ietf.org/doc/html/rfc4648" target="_blank">RFC 4648</a>, which says that the transcoding table </p>
<ul>
<li>starts with capital letters of the English alphabet A-Z, </li>
<li>followed by small letters of the English alphabet a-z, </li>
<li>followed by digits 0-9, and </li>
<li>is concluded by symbols + and /. </li>
</ul>
<p>The padding symbol, not included in the table is the equality sign =.</p>
<h2>Base64 Transcoding Mechanism</h2>
<p>The transcoding mechanism for binaries takes into account that our data exists in byte-sized segments, meaning that whatever data we take and split into segments of 8-bit size, every segment will be full by design (because of the memory addressing schema), without the need to do any padding. </p>
<p>From this point on, we will consider our data on the bit level. </p>
<p>The process is very simple: we take three byte-sized segments and consider them as a segment of 3 x 8 bits = 24 bits. </p>
<p class="has-global-color-8-background-color has-background"><img src="https://s.w.org/images/core/emoji/13.1.0/72x72/1f447.png" alt="?" class="wp-smiley" style="height: 1em; max-height: 1em;" /> This choice of length will be further discussed in our section on <em>“The Least Common Multiple”</em> below.</p>
<p>Our segment of 24 bits is further segmented into four 6-bit segments, which will represent our keys for the transcoding table. Each of the four keys gets transcoded to the associated symbol, i.e. one of the 64 available symbols in the transcoding table. </p>
<p>For instance, if our 6-bit segment is 000 111, it gets transcoded to symbol <code>'H'</code>, and a segment 111 000 gets transcoded to symbol <code>'4'</code>.</p>
<h2>Base64 Special Considerations</h2>
<p>When there is a case where our ending data segment length does not correspond to three bytes, but two bytes or a one-byte segment instead, we have to take into account special considerations, and this is where we introduce bit padding. </p>
<p>First, let us discuss how to process the two-byte segment. </p>
<p>A two-byte segment consists of 16 bits that get split into two whole 6-bit segments (2 x 6 bit = 12 bit) and a rest of 4 bits. These four bits are padded with two 0 bits that will complete the segment to a 6-bit length. </p>
<p>However, now that we have three 6-bit segments, and four segments were originally needed, the transcoding mechanism will take note of that and introduce a special, character-level padding symbol to account for the last segment. </p>
<p>This symbol does not exist in the transcoding table and is denoted as <code>'='</code>. </p>
<p>Here is an example of such a case: the original 2-byte data segment 0011 0101 0010 1010 gets split into 001 101 010 010 101 0+00 where bits 00 are used as segment-level padding to ensure us having three full segments. </p>
<p>These segments are transcoded to symbols <code>'NSo'</code> and the symbol <code>'='</code> is added to account for the missing 6-bit segment, resulting in <code>'NSo='</code>. </p>
<p>Considering the given example, we can always tell that if there is only one symbol <code>'='</code> in the end, two bits were added to the original data and will be removed when we transcode the Base64 string back to the original data. </p>
<p>In the second possible example, our original data is made up of only one byte. This byte will get split into one 6-bit segment and the remaining two bits are padded with four 0 bits that will complete the segment to a full 6-bit length. </p>
<p>Now that we have two 6-bit segments, and four segments were originally needed, the special symbol will be used two times to account for the missing segments: <code>'=='</code>. </p>
<p>Here is an example of such a case: the original 1-byte data segment 0011 0101 gets split into 001 101 01+0 000 where bits 0 000 are used as segment-level padding to ensure us having two full segments. These segments are transcoded to symbols <code>'NQ'</code> and two padding symbols <code>'=='</code> are added to account for the two missing 6-bit segments, resulting in <code>'NQ=='</code>. </p>
<p>Considering the given example, we can always tell that if there are two symbols <code>'='</code> in the end, four bits were added to the original data and will be removed when we transcode the Base64 string back to the original data.</p>
<h2>Base64 Encoder/Decoder Implementation in Python</h2>
<p>A Python implementation of a Base64 encoder/decoder is presented below. Presumably less intuitive parts are commented on for better understanding and convenience.</p>
<pre class="EnlighterJSRAW" data-enlighter-language="python" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">class Base64(object): CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/' # 1-byte padding symbol. padding_symbol = b'\x00' @classmethod def chunk(cls, data, length): # Creates an array of data chunks (segments). data_chunks = [data[i:i + length] for i in range(0, len(data), length)] return data_chunks # Encodes the string to a Base64 string. @classmethod def encode(cls, data): padding_length = 0 # Calculates the length of the padding. if len(data) % 3 != 0: padding_length = (3 - len(data) % 3) % 3 data += cls.padding_symbol * padding_length # Splits the data in three-byte chunks. chunks_3_byte = cls.chunk(data, 3) # Generates a binary string representation of each byte in the chunk, # i.e. three bytes -> three binary representations per chunk. bin_string_repr = "" for chunk in chunks_3_byte: for byte in chunk: # Cuts off the '0b' string prefix and appends # to the 'bin_string_repr'. # {:0>8} stands for leading zeroes, align right, 8 places. bin_string_repr += "{:0>8}".format(bin(byte)[2:]) # Splits the data in six-bit chunks. chunks_6_bit = cls.chunk(bin_string_repr, 6) base64_encoded_str = "" for element in chunks_6_bit: # Transcodes the binary representation (2) string to an integer, # and maps it to an alphanumeric string. base64_encoded_str += cls.CHARS[int(element, 2)] # Encodes the ending with the padding character(s) '='. base64_encoded_str = base64_encoded_str[:-padding_length] + '=' * padding_length return base64_encoded_str @classmethod def decode(cls, data): # Counts the number of '=' occurrences; this way we'll know # how much of the padding we should trim (0, 1 or 2 chars). replaced = data.count('=') # Replaces '=' by 'A'; it will be trimmed at the end, but we # need it until then to retain 3-byte segments. data = data.replace('=', 'A') binstring = '' # Processes each character and returns its binary code. for char in data: # {:0>6b} stands for leading zeroes, align right, 6 places, binary. binstring += "{:0>6b}".format(cls.CHARS.index(char)) # Splits the data in 1-byte (8-bit) chunks. chunks_1_byte = cls.chunk(binstring, 8) base64_decoded_str = b'' for chunk in chunks_1_byte: # Creates the decoded byte-string. base64_decoded_str += bytes([int(chunk, 2)]) return base64_decoded_str[:-replaced] if __name__ == "__main__": b64_enc = Base64.encode(b'Finxter rules!') print(b64_enc) b64_dec = Base64.decode(b64_enc) print(b64_dec)
</pre>
<h2>The Least Common Multiple</h2>
<p>You might have asked yourself, why are we taking exactly 24 bits as a segment for transcoding to Base64? </p>
<p>There is a very simple reason behind this step, and it is called “the least common multiple”, also denoted shortly as “LSM”. </p>
<p class="has-global-color-8-background-color has-background"><strong>Least Common Multiple (LSM):</strong> Given any two numbers A and B, the least common multiple of A and B is the smallest number that is divisible by both A and B without remainders. For instance, the least common multiple of 2 and 3 is 6, because 6 / 2 = 3 with remainder = 0, and 6 / 3 = 2 with remainder = 0. </p>
<p>In the case of our specific interest, numbers A and B represent the length of segments in our addressing schema (8 bits) and the length of a binary represented character in the Base64 transcoding table (6 bits -&gt; 64 characters). </p>
<p>By calculating the least common multiple for 6 and 8, we get 24. </p>
<p>If we perform a validity check: 24 / 6 = 4 with remainder = 0, 24 / 8 = 3 with remainder = 0, we can confirm that 24 indeed is a length that should be used for segment generation to support both 6-bit and 8-bit segments.</p>
<h2>Conclusion</h2>
<p>In this article, we learned about the Base64 transcoding mechanism.&nbsp;</p>
<ul>
<li>First, we explained the use of Base64 in a real-world context.&nbsp;</li>
<li>Second, we touched upon the topic of memory addressing schema.&nbsp;</li>
<li>Third, we got acquainted with the transcoding table.&nbsp;</li>
<li>Fourth, we explained how the transcoding mechanism works.</li>
<li>Fifth, we dove into special considerations on how to handle “incomplete” data (incomplete in terms of not being a 24-bit multiple).</li>
<li>Sixth, we analyzed a Base64 implementation.</li>
<li>Seventh, we held our breath for some simple math theory on the least common multiple.</li>
</ul>
<hr class="wp-block-separator"/>
</div>


https://www.sickgaming.net/blog/2022/04/...ing-video/
Reply



Forum Jump:


Users browsing this thread:
1 Guest(s)

Forum software by © MyBB Theme © iAndrew 2016