This sample contains a block-by-block character set conversion algorithm for a very large files, you can use it for files with any size, this code work also with files over 2GB or 4GB. It takes maximum of 5*BlockSize of memory to convert the data (5MB of memory for the 1MB block).
The SourceCharSet may be only a charset with a constant char size, one, two or four bytes (and the BlockSize must be an integral multiple of the character size). You can use single byte charsets as Windows-1250, windows-1251, Latin II, ibm850... for the source file, but not utf-8 (it has a variable size of char). You can use any destination charset, including utf-8
Charset conversion of very large files - 2GB and more | |
---|---|
Option Explicit FileConvert "H:\giga.dat", "utf-8", "Z:\giga-01.dat", "windows-1250" 'FileConvert "H:\giga.dat", "windows-1250", "Z:\giga-01.dat", "windows-1250" 'Charset conversion of a large file - 2GB, 4GB or more 'ScriptUtilities.ByteArray VBA sample Sub FileConvert(SourceFileName, SourceCharSet, DestFileName, DestinationCharSet) 'Set block size. Bigger block: better speed, more resources. Const BlockSize = &H100000 Dim sPos, dPos, sByteArray, dByteArray, TotalBytes , ReadBlockSize Set sByteArray = CreateObject("ScriptUtils.ByteArray") sByteArray.CharSet = SourceCharSet 'Get a source file length. TotalBytes = GetFileSize(SourceFileName) 'Clear the destination file DeleteFile DestFileName ' If TotalBytes > 0 Then 'Convert only nonzero files. 'For each data block from the source file For sPos = 0 To TotalBytes Step BlockSize Wscript.Echo sPos If BlockSize + sPos > TotalBytes Then 'Last block - read only rest of data. ReadBlockSize = TotalBytes - sPos Else ReadBlockSize = BlockSize End If 'Read the block of source data. sByteArray.ReadFrom SourceFileName, sPos + 1, ReadBlockSize 'Convert the data to the destination charset Set dByteArray = sByteArray.CharSetConvert(DestinationCharSet) ' and store them to a disk. 0 means store the data at the end of file dByteArray.SaveAs DestFileName, 0 'Increment the destination position dPos = dPos + dByteArray.Length Next ' sPos ' End If ' TotalBytes > 0 Then End Sub 'Get file size - 64 bit version, unlimited size Function GetFileSize(FileName) Dim Kernel: Set Kernel = CreateObject("ScriptUtils.Kernel") On Error Resume Next GetFileSize = Kernel.GetFileSize(FileName) If err<>0 Then GetFileSize = 0 On Error Goto 0 End Function Function DeleteFile(FileName) Dim Kernel Set Kernel = CreateObject("ScriptUtils.Kernel") On Error Resume Next Kernel.DeleteFile FileName On Error Goto 0 End Function |
Works with safearray binary data - save/restore binary data from/to a disk, convert to a string/hexstring, codepage/charset conversions, Base64 conversion, etc.
ByteArray is a COM class specially designed to work with Microsoft Windows Scripting engines - VB Script and JScript in Active Server Pages or WSH and in CHM or HTA applications. It also works with VB Net, Visual basic (VBA - VB 5, VB 6, Word, Excel, Access, …), C#, J#, C++, ASP, ASP.Net, Delphi and with T-SQL OLE functions - see Use ByteArray object article. You can also use the object in other programming environments with COM support, such is PowerBuilder.
Source code for ByteArray is available within distribution license, please see License page for ASP file upload and ScriptUtilities.
Huge ASP upload is easy to use, hi-performance ASP file upload component with progress bar indicator. This component lets you upload multiple files with size up to 4GB to a disk or a database along with another form fields. Huge ASP file upload is a most featured upload component on a market with competitive price and a great performance . The software has also a free version of asp upload with progress, called Pure asp upload , written in plain VBS, without components (so you do not need to install anything on server). This installation package contains also ScriptUtilities library. Script Utilities lets you create hi-performance log files , works with binary data , you can download multiple files with zip/arj compression, work with INI files and much more with the ASP utility.
© 1996 - 2011 Antonin Foller, Motobit Software | About, Contacts | e-mail: info@pstruh.cz