tag:help.rubygems.org,2010-01-19:/discussions/problems/501-broken-utf-8-handling-in-newest-rubygems-when-environment-locales-are-not-setRubyGems.org: Discussion 2011-11-21T10:12:05Ztag:help.rubygems.org,2010-01-19:Comment/56058482011-02-27T01:05:08Z2011-02-27T01:05:09ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Hi,</p>
<p>I'm encountering some problems after upgrade to newest
Rubygems.</p>
<p>It happens when locale settings from the environment are empty
(not even "C" locale) and someone tries to install some gem that
was created using the newest Rubygems. The major precondition for
error to occur is a UTF-8 character in some descriptive field (e.g.
developer). The locale settings used when building a gem are
irrevelant.</p>
<p>rubygems-update (1.5.2)<br>
gemcutter (0.6.1)</p>
<p>Steps to reproduce:</p>
<pre>
<code>unset LANG
unset LC_ALL
gem install i18n-inflector
ERROR: While executing gem ... (ArgumentError)
invalid byte sequence in US-ASCII</code>
</pre>
<p>But the same package built one day before (before Rubygems
upgrade):</p>
<pre>
<code>unset LANG
unset LC_ALL
gem install i18n-inflector --version 2.5.0
Fetching: i18n-inflector-2.5.0.gem (100%)
Successfully installed i18n-inflector-2.5.0
1 gem installed</code>
</pre>
<p>Regards,<br>
Paweł</p></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-02-27T18:29:05Z2011-02-27T18:29:05ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Wow, strange...not sure what's up here, the guys that work on
the RG client library will most likely have a better idea.</p>
<p>What OS is this on? Can you show us your <code>gem env</code>
output?</p></div>Nick Quarantotag:help.rubygems.org,2010-01-19:Comment/56058482011-02-28T00:38:41Z2011-02-28T00:38:43ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Hi,</p>
<p>The problem is OS-independent. It happened on Mac OS X Snow
Leopard and on Ubuntu Server. The problem was detected on Ruby
1.9.</p>
<p>BTW, prerequisites to trigger the error when installing a
gem:</p>
<pre>
<code>unset LANG
unset LC_ALL
unset LC_COLLATE
unset LC_CTYPE
unset LC_MESSAGES
unset LC_MONETARY
unset LC_NUMERIC
unset LC_TIME
locale
# => LANG=
# => LC_COLLATE="C"
# => LC_CTYPE="C"
# => LC_MESSAGES="C"
# => LC_MONETARY="C"
# => LC_NUMERIC="C"
# => LC_TIME="C"
# => LC_ALL=</code>
</pre>
<p>First failing env:</p>
<pre>
<code>RubyGems Environment:
- RUBYGEMS VERSION: 1.5.2
- RUBY VERSION: 1.9.2 (2011-02-18 patchlevel 180) [x86_64-darwin10.6.0]
- INSTALLATION DIRECTORY: /usr/local/Cellar/ruby/1.9.2-p180/lib/ruby/gems/1.9.1
- RUBY EXECUTABLE: /usr/local/Cellar/ruby/1.9.2-p180/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/Cellar/ruby/1.9.2-p180/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-10
- GEM PATHS:
- /usr/local/Cellar/ruby/1.9.2-p180/lib/ruby/gems/1.9.1
- /var/root/.gem/ruby/1.9.1
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- :sources => ["http://gems.rubyforge.org/", "http://gems.github.com/"]
- REMOTE SOURCES:
- http://gems.rubyforge.org/
- http://gems.github.com/</code>
</pre>
<p>Second failing env:<br></p>
<pre>
<code>RubyGems Environment:
- RUBYGEMS VERSION: 1.5.2
- RUBY VERSION: 1.9.2 (2010-08-18 patchlevel 0) [x86_64-darwin10.6.0]
- INSTALLATION DIRECTORY: /usr/local/rvm/gems/ruby-1.9.2-p0
- RUBY EXECUTABLE: /usr/local/rvm/rubies/ruby-1.9.2-p0/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/rvm/gems/ruby-1.9.2-p0/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-10
- GEM PATHS:
- /usr/local/rvm/gems/ruby-1.9.2-p0
- /usr/local/rvm/gems/ruby-1.9.2-p0@global
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://rubygems.org/</code>
</pre>
<p>RubyGems 1.5.2 on Ruby 1.8 seems unaffected (but since 1.8 is
often encoding-unaware<br>
it might be just a bug covering other bug).</p>
<p>Note, that this error occurs when installing packages
containing<br>
UTF-8 characters in some fields that were built with RubyGems
1.5.2.<br>
Packages built using previous version of RubyGems are installed
successully.</p>
<p>There is a difference in specfiles, that might help in tracking
cause<br>
of this problem. Package i18n-inflector-2.5.0 was build with old
RG,<br>
package i18n-inflector-2.5.1 with new RG. Both were installed
using<br>
new RG, but in case of i18n-inflector-2.5.1 installation failed<br>
with the error quoted in my previous post.</p>
<p>Look:</p>
<pre>
<code> $ diff i18n-inflector-2.5.0.gemspec i18n-inflector-2.5.1.gemspec</code>
</pre>
<p>Result: <a href=
"http://pastie.org/1615020">http://pastie.org/1615020</a></p>
<p>(I used Pastie since this markup makes backslashes with
character codes interpretable)</p>
<p>See: the <code>authors</code> line. New RubyGems produce
unescaped version.<br>
I don't know, maybe that's intended, but the new RG client<br>
has some problems with installing such a gem when environment<br>
locales are not set.</p>
<p>Regards,<br>
Paweł</p></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-03-06T23:04:57Z2011-03-06T23:04:58ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>It seems that exists in many previous RG versions and it's not a
bug of RG itself but it would be nice to have some workaround for
that in RG.</p></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-05-27T00:29:17Z2011-05-27T00:29:17ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Is this still happening in later versions? Digging up open
tickets, I haven't heard or seen other reports of this.</p></div>Nick Quarantotag:help.rubygems.org,2010-01-19:Comment/56058482011-05-27T22:34:16Z2011-05-27T22:34:17ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>I got this today when using "pusher-gem" (<a href=
"https://github.com/pusher/pusher-gem">https://github.com/pusher/pusher-gem</a>)
on Ruby 1.9.2-p180 and RG 1.8.4</p></div>Grzegorz Kazulaktag:help.rubygems.org,2010-01-19:Comment/56058482011-06-19T16:07:36Z2011-06-19T16:13:46ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Hello,</p>
<p>Can you guys open a bug report / issue for RubyGems itself:</p>
<p><a href=
"https://github.com/rubygems/rubygems/issues">https://github.com/rubygems/rubygems/issues</a></p>
<p>For us to better work on fixes for RubyGems that is the right
place.</p>
<p>Thank you.</p></div>Luis Lavenatag:help.rubygems.org,2010-01-19:Comment/56058482011-06-19T16:14:13Z2011-06-19T16:14:13ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Sorry, by mistake I provided the wrong link to RubyGems issue
tracker.</p>
<p>Please use this: <a href=
"https://github.com/rubygems/rubygems/issues">https://github.com/rubygems/rubygems/issues</a></p>
<p>Thank you.</p></div>Luis Lavenatag:help.rubygems.org,2010-01-19:Comment/56058482011-08-10T08:57:53Z2011-08-10T08:57:54ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Use following command to fix the problem</p>
<pre>
<code>export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
± locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"</code>
</pre></div>zorastag:help.rubygems.org,2010-01-19:Comment/56058482011-08-10T09:32:26Z2011-08-10T09:32:28ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Since I am not in control of systems that are using my gem this
workaround cannot be applied by me.</p>
<p>Yesterday I've discovered that it may be related to psych
parser. When using syck parser whlie building gem the problem
dissapears during installation since UTF-8 characters are escaped
in a gem manifest file.</p>
<p>To apply the workaround the developer has to set the YAML engine
to syck in the Rakefile and/or gemspec:</p>
<pre>
<code> if !defined?(YAML::ENGINE).nil? && YAML::ENGINE.respond_to?(:yamler)
YAML::ENGINE.yamler = 'syck'
end</code>
</pre>
<p>See also:</p>
<ul>
<li><a href=
"https://github.com/siefca/i18n-inflector/issues/3">https://github.com/siefca/i18n-inflector/issues/3</a></li>
<li><a href=
"https://github.com/rubygems/rubygems/issues/139">https://github.com/rubygems/rubygems/issues/139</a></li>
</ul></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-10-10T17:43:22Z2011-10-10T17:43:22ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>This is fixed.</p></div>Eric Hodel