Detecting a code page from text is a very tricky task. Similarly, the IMultiLang2 interface has a function to detect the encoding of an incoming byte array. This is very handy for codepage detection of text stored in files or for text that needs to be sent over the internet. The EncodingTools class offers some easy-to-use functions to determine the best encoding for different scenarios. I started this along with another component that constructs MIME conformant emails. The body of the email is passed as String.
The user had to provide the charset to use for the Transfer-Encoding by hand. This is fine as long as you know the target character set or always assume Unicode. But it is definitely not a good solution if you have an end-user GUI application most users do not even know what an "encoding" is.
This is not only ugly, it does not even work properly. All single byte encodings are binary equal in their encoding result. The codepage is only used to map single bytes to the correct character for display.
Then I remembered the IMultiLang2. DetectInputCodepage method that was introduced along with Internet Explorer 5. This method detects the encoding used in a text used by Internet Explorer to do automatic codepage detection if the header is missing from a page. Even this was not suitable for my problem, and I wondered if there had been development since version 5. Internet Explorer 5. The MLang. Remove From My Forums. Answered by:.
Archived Forums. Visual C. Sign in to vote. ReadLine ; System. CurrentEncoding; Console. WriteLine enc. Thanks for your helping,. Wednesday, June 22, AM. Thursday, June 23, AM. CurrentEncoding ; " It can only indicate that the encoding of stream,not the encoding of the file. There are, however, other ways to detect the encoding. If the data has a length that's a multiple of 4, and follows one of these patterns, you can safely assume it's UTF False positives are nearly impossible due to the rarity of 00 bytes in byte-oriented encodings.
No BOM, but you don't need one. But you can't rely on this. False positives are rare. Specifically, given that the data is not ASCII, the false positive rate for a 2-byte sequence is only 3.
For a byte sequence, it's less than 0. For a byte sequence, it's less than 1 in a million. If you happen to have a file that consists mainly of ISO characters, having half of the file's bytes be 00 would also be a strong indicator of UTF If your file starts with the bytes 3C 3F 78 6D 6C i.
If present, then use that encoding. In general, if you have a file format that contains an encoding declaration, then look for that declaration rather than trying to guess the encoding. There are hundreds of other encodings, which require more effort to detect. I recommend trying Mozilla's charset detector or a. NET port of it. If you've ruled out the UTF encodings, and don't have an encoding declaration or statistical detection that points to a different encoding, assume ISO or the closely related Windows Being Windows' default code page for English and other popular languages like Spanish, Portuguese, German, and French , it's the most commonly encountered encoding other than UTF It does the BOM detection automatically first, and then tries to differentiate between Unicode encodings without BOM, vs some other default encoding generally Windows, incorrectly labelled as Encoding.
ASCII in. Here is my code that detects all encodings that Microsoft detects in Framework 4 in the StreamReader class. Obviously you must call this function immediately after opening the stream before reading anything else from the stream because the BOM are the first bytes in the stream. This function requires a Stream that can seek for example a FileStream. If you have a Stream that cannot seek you must write a more complicated code that returns a Byte buffer with the bytes that have already been read but that are not BOM.
It is easy to use and gives some really good results. If your file starts with the bytes 60, , 56, 46 and 49, then you have an ambiguous case.
This function opens the file an determines the Encoding by the BOM. GetString Encoding. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. How to detect the character encoding of a text file?
Ask Question. Asked 11 years ago. Active 3 months ago. Viewed k times. I try to detect which character encoding is used in my file. Open ; file. Read buffer, 0, 5 ; file. Is there a chart that shows which encoding matches those five first bytes? Jon k 74 74 gold badges silver badges bronze badges.
The byte order mark should not be used to detect encodings. The BOM should only be used to detect byte order hence its name. Also UTF-8 strictly speaking should not even have a byte order mark and adding one can interfere with some software that doesn't expect it. Mark Bayers, so it's there a way i can detech witch encoding are use in my file? Also, the BOM in theory should indicate as you say, but in practice, it acts as a signature to show what encoding it.
See: unicode. The BOM is a bullet proof way to detect encoding.
0コメント