tag:help.rubygems.org,2010-01-19:/discussions/problems/501-broken-utf-8-handling-in-newest-rubygems-when-environment-locales-are-not-setRubyGems.org: Discussion 2011-11-21T10:12:05Ztag:help.rubygems.org,2010-01-19:Comment/56058482011-02-27T18:29:05Z2011-02-27T18:29:05ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Wow, strange...not sure what's up here, the guys that work on
the RG client library will most likely have a better idea.</p>
<p>What OS is this on? Can you show us your <code>gem env</code>
output?</p></div>Nick Quarantotag:help.rubygems.org,2010-01-19:Comment/56058482011-02-28T00:38:41Z2011-02-28T00:38:43ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Hi,</p>
<p>The problem is OS-independent. It happened on Mac OS X Snow
Leopard and on Ubuntu Server. The problem was detected on Ruby
1.9.</p>
<p>BTW, prerequisites to trigger the error when installing a
gem:</p>
<pre>
<code>unset LANG
unset LC_ALL
unset LC_COLLATE
unset LC_CTYPE
unset LC_MESSAGES
unset LC_MONETARY
unset LC_NUMERIC
unset LC_TIME
locale
# => LANG=
# => LC_COLLATE="C"
# => LC_CTYPE="C"
# => LC_MESSAGES="C"
# => LC_MONETARY="C"
# => LC_NUMERIC="C"
# => LC_TIME="C"
# => LC_ALL=</code>
</pre>
<p>First failing env:</p>
<pre>
<code>RubyGems Environment:
- RUBYGEMS VERSION: 1.5.2
- RUBY VERSION: 1.9.2 (2011-02-18 patchlevel 180) [x86_64-darwin10.6.0]
- INSTALLATION DIRECTORY: /usr/local/Cellar/ruby/1.9.2-p180/lib/ruby/gems/1.9.1
- RUBY EXECUTABLE: /usr/local/Cellar/ruby/1.9.2-p180/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/Cellar/ruby/1.9.2-p180/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-10
- GEM PATHS:
- /usr/local/Cellar/ruby/1.9.2-p180/lib/ruby/gems/1.9.1
- /var/root/.gem/ruby/1.9.1
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- :sources => ["http://gems.rubyforge.org/", "http://gems.github.com/"]
- REMOTE SOURCES:
- http://gems.rubyforge.org/
- http://gems.github.com/</code>
</pre>
<p>Second failing env:<br></p>
<pre>
<code>RubyGems Environment:
- RUBYGEMS VERSION: 1.5.2
- RUBY VERSION: 1.9.2 (2010-08-18 patchlevel 0) [x86_64-darwin10.6.0]
- INSTALLATION DIRECTORY: /usr/local/rvm/gems/ruby-1.9.2-p0
- RUBY EXECUTABLE: /usr/local/rvm/rubies/ruby-1.9.2-p0/bin/ruby
- EXECUTABLE DIRECTORY: /usr/local/rvm/gems/ruby-1.9.2-p0/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-10
- GEM PATHS:
- /usr/local/rvm/gems/ruby-1.9.2-p0
- /usr/local/rvm/gems/ruby-1.9.2-p0@global
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://rubygems.org/</code>
</pre>
<p>RubyGems 1.5.2 on Ruby 1.8 seems unaffected (but since 1.8 is
often encoding-unaware<br>
it might be just a bug covering other bug).</p>
<p>Note, that this error occurs when installing packages
containing<br>
UTF-8 characters in some fields that were built with RubyGems
1.5.2.<br>
Packages built using previous version of RubyGems are installed
successully.</p>
<p>There is a difference in specfiles, that might help in tracking
cause<br>
of this problem. Package i18n-inflector-2.5.0 was build with old
RG,<br>
package i18n-inflector-2.5.1 with new RG. Both were installed
using<br>
new RG, but in case of i18n-inflector-2.5.1 installation failed<br>
with the error quoted in my previous post.</p>
<p>Look:</p>
<pre>
<code> $ diff i18n-inflector-2.5.0.gemspec i18n-inflector-2.5.1.gemspec</code>
</pre>
<p>Result: <a href=
"http://pastie.org/1615020">http://pastie.org/1615020</a></p>
<p>(I used Pastie since this markup makes backslashes with
character codes interpretable)</p>
<p>See: the <code>authors</code> line. New RubyGems produce
unescaped version.<br>
I don't know, maybe that's intended, but the new RG client<br>
has some problems with installing such a gem when environment<br>
locales are not set.</p>
<p>Regards,<br>
Paweł</p></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-03-06T23:04:57Z2011-03-06T23:04:58ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>It seems that exists in many previous RG versions and it's not a
bug of RG itself but it would be nice to have some workaround for
that in RG.</p></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-05-27T00:29:17Z2011-05-27T00:29:17ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Is this still happening in later versions? Digging up open
tickets, I haven't heard or seen other reports of this.</p></div>Nick Quarantotag:help.rubygems.org,2010-01-19:Comment/56058482011-05-27T22:34:16Z2011-05-27T22:34:17ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>I got this today when using "pusher-gem" (<a href=
"https://github.com/pusher/pusher-gem">https://github.com/pusher/pusher-gem</a>)
on Ruby 1.9.2-p180 and RG 1.8.4</p></div>Grzegorz Kazulaktag:help.rubygems.org,2010-01-19:Comment/56058482011-06-19T16:07:36Z2011-06-19T16:13:46ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Hello,</p>
<p>Can you guys open a bug report / issue for RubyGems itself:</p>
<p><a href=
"https://github.com/rubygems/rubygems/issues">https://github.com/rubygems/rubygems/issues</a></p>
<p>For us to better work on fixes for RubyGems that is the right
place.</p>
<p>Thank you.</p></div>Luis Lavenatag:help.rubygems.org,2010-01-19:Comment/56058482011-06-19T16:14:13Z2011-06-19T16:14:13ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Sorry, by mistake I provided the wrong link to RubyGems issue
tracker.</p>
<p>Please use this: <a href=
"https://github.com/rubygems/rubygems/issues">https://github.com/rubygems/rubygems/issues</a></p>
<p>Thank you.</p></div>Luis Lavenatag:help.rubygems.org,2010-01-19:Comment/56058482011-08-10T08:57:53Z2011-08-10T08:57:54ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Use following command to fix the problem</p>
<pre>
<code>export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
± locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"</code>
</pre></div>zorastag:help.rubygems.org,2010-01-19:Comment/56058482011-08-10T09:32:26Z2011-08-10T09:32:28ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>Since I am not in control of systems that are using my gem this
workaround cannot be applied by me.</p>
<p>Yesterday I've discovered that it may be related to psych
parser. When using syck parser whlie building gem the problem
dissapears during installation since UTF-8 characters are escaped
in a gem manifest file.</p>
<p>To apply the workaround the developer has to set the YAML engine
to syck in the Rakefile and/or gemspec:</p>
<pre>
<code> if !defined?(YAML::ENGINE).nil? && YAML::ENGINE.respond_to?(:yamler)
YAML::ENGINE.yamler = 'syck'
end</code>
</pre>
<p>See also:</p>
<ul>
<li><a href=
"https://github.com/siefca/i18n-inflector/issues/3">https://github.com/siefca/i18n-inflector/issues/3</a></li>
<li><a href=
"https://github.com/rubygems/rubygems/issues/139">https://github.com/rubygems/rubygems/issues/139</a></li>
</ul></div>Paweł Wilktag:help.rubygems.org,2010-01-19:Comment/56058482011-10-10T17:43:22Z2011-10-10T17:43:22ZBroken UTF-8 handling in newest Rubygems when environment locales are not set<div><p>This is fixed.</p></div>Eric Hodel